Re: Data Import Blocker - Solr

[Shawn Heisey]

On 12/18/2020 12:03 AM, basel altameme wrote:

While trying to Import & Index data from MySQL DB custom view i am facing the 
error below:
Data Config problem: The value of attribute "query" associated with an element type 
"entity" must not contain the '<' character.
Please note that in my SQL statements i am using '<>' as an operator for 
comparing only.
sample line:
         when (`v`.`live_type_id` <> 1) then 100


These configurations are written in XML.  So you must encode the 
character using XML-friendly notation.


Instead of <> it should say  to be correct.  Or you could use != 
which is also correct SQL notation for "not equal to".


Thanks,
Shawn

Re: Data Import Blocker - Solr

[Erick Erickson]

Have you tried escaping that character?

> On Dec 18, 2020, at 2:03 AM, basel altameme  
> wrote:
> 
> Dear,
> While trying to Import & Index data from MySQL DB custom view i am facing the 
> error below:
> Data Config problem: The value of attribute "query" associated with an 
> element type "entity" must not contain the '<' character.
> Please note that in my SQL statements i am using '<>' as an operator for 
> comparing only.
> sample line:
> when (`v`.`live_type_id` <> 1) then 100
> 
> Kindly advice.
> Regards,Basel
> 

Data Import Blocker - Solr

[basel altameme]

Dear,
While trying to Import & Index data from MySQL DB custom view i am facing the 
error below:
Data Config problem: The value of attribute "query" associated with an element 
type "entity" must not contain the '<' character.
Please note that in my SQL statements i am using '<>' as an operator for 
comparing only.
sample line:
        when (`v`.`live_type_id` <> 1) then 100

Kindly advice.
Regards,Basel

Re: CSV import to SOLR

[Alexandre Rafalovitch]

UpdateRequestProcessor is probably your best bet, because you can
choose to trigger them by name with import parameter.

Then you can probably use CloneFieldUpdateProcessorFactory.

Regards,
   Alex.

http://www.solr-start.com/ - Resources for Solr users, new and experienced


On 27 September 2017 at 13:33, Zisis Simaioforidis  wrote:
> So there is no way of telling SOLR to duplicate a column of CSV by just
> using some parameters during the import request?
>
> Just for the CSV.
>
> The truth is  copyfield crosseb my mind but it's just too brute force
> because it will affect all documents imported. And CSV is NOT the only
> method we are importing. We also use MARC and XML.
>
> Zisis
>
>
>
> Στις 27/9/2017 5:52 PM, ο Erick Erickson έγραψε:
>>
>> If you always want to do this exact thing, it looks like a copyField
>> directive in your schema.
>>
>> If it has to be more nuanced, you can use something like
>> StatelessScriptUpdateProcessorFactory.
>>
>> Both of these would affect _all_ documents coming in to Solr, so may
>> be too blunt a hammer.
>>
>> Best,
>> Erick
>>
>> On Wed, Sep 27, 2017 at 3:07 AM, Zisis Simaioforidis 
>> wrote:
>>>
>>> Is there a way to map a field value based on another field value without
>>> replicatiing the columns in the CSV itself?
>>>
>>> for example i tried : literal.title_fullStr=f.title_short but it doesn't
>>> seem to work.
>>>
>>> Thank you
>>>
>

Re: CSV import to SOLR

[Zisis Simaioforidis]

So there is no way of telling SOLR to duplicate a column of CSV by just 
using some parameters during the import request?


Just for the CSV.

The truth is  copyfield crosseb my mind but it's just too brute force 
because it will affect all documents imported. And CSV is NOT the only 
method we are importing. We also use MARC and XML.


Zisis


Στις 27/9/2017 5:52 PM, ο Erick Erickson έγραψε:

If you always want to do this exact thing, it looks like a copyField
directive in your schema.

If it has to be more nuanced, you can use something like
StatelessScriptUpdateProcessorFactory.

Both of these would affect _all_ documents coming in to Solr, so may
be too blunt a hammer.

Best,
Erick

On Wed, Sep 27, 2017 at 3:07 AM, Zisis Simaioforidis  wrote:

Is there a way to map a field value based on another field value without
replicatiing the columns in the CSV itself?

for example i tried : literal.title_fullStr=f.title_short but it doesn't
seem to work.

Thank you

Re: CSV import to SOLR

[Erick Erickson]

If you always want to do this exact thing, it looks like a copyField
directive in your schema.

If it has to be more nuanced, you can use something like
StatelessScriptUpdateProcessorFactory.

Both of these would affect _all_ documents coming in to Solr, so may
be too blunt a hammer.

Best,
Erick

On Wed, Sep 27, 2017 at 3:07 AM, Zisis Simaioforidis  wrote:
> Is there a way to map a field value based on another field value without
> replicatiing the columns in the CSV itself?
>
> for example i tried : literal.title_fullStr=f.title_short but it doesn't
> seem to work.
>
> Thank you
>

CSV import to SOLR

[Zisis Simaioforidis]

Is there a way to map a field value based on another field value without 
replicatiing the columns in the CSV itself?


for example i tried : literal.title_fullStr=f.title_short but it doesn't 
seem to work.


Thank you

Re: Classes in solr_home /lib cannot import from solr/dist

[Callum Lamb]

Good to know Solr already loads them, that removed a bunch of lines from my
solrconfig.xml.

Having to copy the required jars from dist/ to lib/ isn't ideal but if
that's the only solution then at least I can stop searching for a solution
and figure out how best to deal with this limitation.

I assume the reason for this is that the libs in solr.home.home/lib are
loaded at runtime? I don't know much about how this works in Java but i'm
guessing Solr can access the classes in the Jars but not the other way
around?

Thanks for your help guys.

On Thu, Jan 14, 2016 at 5:03 PM, Shawn Heisey  wrote:

> On 1/14/2016 5:36 AM, Callum Lamb wrote:
> > I've got an extension jar that contains a class which extends from
> >
> > org.apache.solr.handler.dataimport.DataSource
> >
> > But it only works if it's within the solr/dist folder. However when
> stored
> > in the lib/ folder within Solr home. When it tries to load the class it
> > cannot find it's parent:
> >
> > Exception in thread "Thread-69" java.lang.NoClassDefFoundError:
> > org/apache/solr/handler/dataimport/DataSource
> > at
> >
> org.apache.solr.handler.dataimport.DataImporter.getDataSourceInstance(DataImporter.java:374)
> > at
> >
> org.apache.solr.handler.dataimport.ContextImpl.getDataSource(ContextImpl.java:102)
> > Caused by: java.lang.ClassNotFoundException:
> > org.apache.solr.handler.dataimport.DataSource
> >
> > The classes in the lib folder don't have access to the class within the
> > dist folder in their classpath when they are loaded.
> >
> > I'd like the keep my solr install separate from my
> configs/plugins/indexes
> > so I want to avoid putting it into the dist folder unless I absolutely
> have
> > to.
>
> If you're going to put jars in $SOLR_HOME/lib, then you should *only*
> put jars in that directory, and NOT load jars explicitly.  The 
> directives should not be used in solrconfig.xml when jars are loaded
> from this directory, because Solr will automatically load jars from this
> location and make them available to all cores.
>
> If moving all your extra jars (including things like the dataimport jar)
> to $SOLR_HOME/lib and taking out jar loading in solrconfig.xml doesn't
> help, then depending on the Solr version, you *might* be running into
> SOLR-6188.
>
> https://issues.apache.org/jira/browse/SOLR-6188
>
> You'll want to be sure that you don't the same jar more than once.  This
> is the root of the specific problem that SOLR-6188 solves.  Loading the
> same jar more than once can also happen if the jar is in the lib
> directory AND mentioned on a  config element.
>
> Thanks,
> Shawn
>
>

-- 

Mintel Group Ltd | 11 Pilgrim Street | London | EC4V 6RN
Registered in England: Number 1475918. | VAT Number: GB 232 9342 72

Contact details for our other offices can be found at 
http://www.mintel.com/office-locations.

This email and any attachments may include content that is confidential, 
privileged 
or otherwise protected under applicable law. Unauthorised disclosure, 
copying, distribution 
or use of the contents is prohibited and may be unlawful. If you have 
received this email in error,
including without appropriate authorisation, then please reply to the 
sender about the error 
and delete this email and any attachments.

Re: Classes in solr_home /lib cannot import from solr/dist

[Shawn Heisey]

On 1/15/2016 5:36 AM, Callum Lamb wrote:
> Good to know Solr already loads them, that removed a bunch of lines from my
> solrconfig.xml.
>
> Having to copy the required jars from dist/ to lib/ isn't ideal but if
> that's the only solution then at least I can stop searching for a solution
> and figure out how best to deal with this limitation.
>
> I assume the reason for this is that the libs in solr.home.home/lib are
> loaded at runtime? I don't know much about how this works in Java but i'm
> guessing Solr can access the classes in the Jars but not the other way
> around?

Classloaders in Java are a complex topic that I do not fully understand.

The contents of the $SOLR_HOME/lib directory are loaded by the main Solr
classloader before any cores are started, and all of the cores that get
started afterwards are able to use those classes.  If the core-level
classloader chooses to load one of the jars a second time, then there
can be problems.

Rather than try and understand all the complexities of class loading, I
find it better to simply place all the jars in the one location that
Solr loads automatically and take the decision away from the individual
cores.  It makes everything easier.

Thanks,
Shawn

Classes in solr_home /lib cannot import from solr/dist

[Callum Lamb]

I've got an extension jar that contains a class which extends from

org.apache.solr.handler.dataimport.DataSource

But it only works if it's within the solr/dist folder. However when stored
in the lib/ folder within Solr home. When it tries to load the class it
cannot find it's parent:

Exception in thread "Thread-69" java.lang.NoClassDefFoundError:
org/apache/solr/handler/dataimport/DataSource
at
org.apache.solr.handler.dataimport.DataImporter.getDataSourceInstance(DataImporter.java:374)
at
org.apache.solr.handler.dataimport.ContextImpl.getDataSource(ContextImpl.java:102)
Caused by: java.lang.ClassNotFoundException:
org.apache.solr.handler.dataimport.DataSource

The classes in the lib folder don't have access to the class within the
dist folder in their classpath when they are loaded.

I'd like the keep my solr install separate from my configs/plugins/indexes
so I want to avoid putting it into the dist folder unless I absolutely have
to.

Is this by design? Is there some kind of configuration somewhere I can
tweak to get this to work?

Cheers,

Callum L.

-- 

Mintel Group Ltd | 11 Pilgrim Street | London | EC4V 6RN
Registered in England: Number 1475918. | VAT Number: GB 232 9342 72

Contact details for our other offices can be found at 
http://www.mintel.com/office-locations.

This email and any attachments may include content that is confidential, 
privileged 
or otherwise protected under applicable law. Unauthorised disclosure, 
copying, distribution 
or use of the contents is prohibited and may be unlawful. If you have 
received this email in error,
including without appropriate authorisation, then please reply to the 
sender about the error 
and delete this email and any attachments.

Re: Classes in solr_home /lib cannot import from solr/dist

[sara hajili]

hi Callum.
you can create a directory for your jar file any where,and u must set jar
file location in  tag in solrConfig.xml
and be carefull that add your lib location at the end of the solr config
default  tag,
because some times your jar need class that at first solr must be load own
class after that load your jar to don't face a class not found exception.


On Thu, Jan 14, 2016 at 4:36 AM, Callum Lamb  wrote:

> I've got an extension jar that contains a class which extends from
>
> org.apache.solr.handler.dataimport.DataSource
>
> But it only works if it's within the solr/dist folder. However when stored
> in the lib/ folder within Solr home. When it tries to load the class it
> cannot find it's parent:
>
> Exception in thread "Thread-69" java.lang.NoClassDefFoundError:
> org/apache/solr/handler/dataimport/DataSource
> at
>
> org.apache.solr.handler.dataimport.DataImporter.getDataSourceInstance(DataImporter.java:374)
> at
>
> org.apache.solr.handler.dataimport.ContextImpl.getDataSource(ContextImpl.java:102)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.solr.handler.dataimport.DataSource
>
> The classes in the lib folder don't have access to the class within the
> dist folder in their classpath when they are loaded.
>
> I'd like the keep my solr install separate from my configs/plugins/indexes
> so I want to avoid putting it into the dist folder unless I absolutely have
> to.
>
> Is this by design? Is there some kind of configuration somewhere I can
> tweak to get this to work?
>
> Cheers,
>
> Callum L.
>
> --
>
> Mintel Group Ltd | 11 Pilgrim Street | London | EC4V 6RN
> Registered in England: Number 1475918. | VAT Number: GB 232 9342 72
>
> Contact details for our other offices can be found at
> http://www.mintel.com/office-locations.
>
> This email and any attachments may include content that is confidential,
> privileged
> or otherwise protected under applicable law. Unauthorised disclosure,
> copying, distribution
> or use of the contents is prohibited and may be unlawful. If you have
> received this email in error,
> including without appropriate authorisation, then please reply to the
> sender about the error
> and delete this email and any attachments.
>
>

Re: Classes in solr_home /lib cannot import from solr/dist

[Callum Lamb]

That's what I did:

My solrconfig.xml has the following (i've hardcoded the version numbers for
now to get regexes out of the picture):




No warning's whatsoever for not finding the jars. And the jars themselves
are in the right order (the second depends on the first).

If i move the data import handler jar to the ${solr.solr.home}/lib/ folder
then everything works. This implies that the solr-dataimporthandler jar
isn't being included properly but I've checked so many times that it's
correct. I can do a full absolute path without the use of solr.install.dir
and solr.solr.home and it still does not work.

The permissions and ownership on the jar files are identical for the 2
jars, if it can load one then it should be able to load the other.




On Thu, Jan 14, 2016 at 2:19 PM, sara hajili  wrote:

> hi Callum.
> you can create a directory for your jar file any where,and u must set jar
> file location in  tag in solrConfig.xml
> and be carefull that add your lib location at the end of the solr config
> default  tag,
> because some times your jar need class that at first solr must be load own
> class after that load your jar to don't face a class not found exception.
>
>
> On Thu, Jan 14, 2016 at 4:36 AM, Callum Lamb  wrote:
>
> > I've got an extension jar that contains a class which extends from
> >
> > org.apache.solr.handler.dataimport.DataSource
> >
> > But it only works if it's within the solr/dist folder. However when
> stored
> > in the lib/ folder within Solr home. When it tries to load the class it
> > cannot find it's parent:
> >
> > Exception in thread "Thread-69" java.lang.NoClassDefFoundError:
> > org/apache/solr/handler/dataimport/DataSource
> > at
> >
> >
> org.apache.solr.handler.dataimport.DataImporter.getDataSourceInstance(DataImporter.java:374)
> > at
> >
> >
> org.apache.solr.handler.dataimport.ContextImpl.getDataSource(ContextImpl.java:102)
> > Caused by: java.lang.ClassNotFoundException:
> > org.apache.solr.handler.dataimport.DataSource
> >
> > The classes in the lib folder don't have access to the class within the
> > dist folder in their classpath when they are loaded.
> >
> > I'd like the keep my solr install separate from my
> configs/plugins/indexes
> > so I want to avoid putting it into the dist folder unless I absolutely
> have
> > to.
> >
> > Is this by design? Is there some kind of configuration somewhere I can
> > tweak to get this to work?
> >
> > Cheers,
> >
> > Callum L.
> >
> > --
> >
> > Mintel Group Ltd | 11 Pilgrim Street | London | EC4V 6RN
> > Registered in England: Number 1475918. | VAT Number: GB 232 9342 72
> >
> > Contact details for our other offices can be found at
> > http://www.mintel.com/office-locations.
> >
> > This email and any attachments may include content that is confidential,
> > privileged
> > or otherwise protected under applicable law. Unauthorised disclosure,
> > copying, distribution
> > or use of the contents is prohibited and may be unlawful. If you have
> > received this email in error,
> > including without appropriate authorisation, then please reply to the
> > sender about the error
> > and delete this email and any attachments.
> >
> >
>

-- 

Mintel Group Ltd | 11 Pilgrim Street | London | EC4V 6RN
Registered in England: Number 1475918. | VAT Number: GB 232 9342 72

Contact details for our other offices can be found at 
http://www.mintel.com/office-locations.

This email and any attachments may include content that is confidential, 
privileged 
or otherwise protected under applicable law. Unauthorised disclosure, 
copying, distribution 
or use of the contents is prohibited and may be unlawful. If you have 
received this email in error,
including without appropriate authorisation, then please reply to the 
sender about the error 
and delete this email and any attachments.

Re: Classes in solr_home /lib cannot import from solr/dist

[Shawn Heisey]

On 1/14/2016 5:36 AM, Callum Lamb wrote:
> I've got an extension jar that contains a class which extends from
>
> org.apache.solr.handler.dataimport.DataSource
>
> But it only works if it's within the solr/dist folder. However when stored
> in the lib/ folder within Solr home. When it tries to load the class it
> cannot find it's parent:
>
> Exception in thread "Thread-69" java.lang.NoClassDefFoundError:
> org/apache/solr/handler/dataimport/DataSource
> at
> org.apache.solr.handler.dataimport.DataImporter.getDataSourceInstance(DataImporter.java:374)
> at
> org.apache.solr.handler.dataimport.ContextImpl.getDataSource(ContextImpl.java:102)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.solr.handler.dataimport.DataSource
>
> The classes in the lib folder don't have access to the class within the
> dist folder in their classpath when they are loaded.
>
> I'd like the keep my solr install separate from my configs/plugins/indexes
> so I want to avoid putting it into the dist folder unless I absolutely have
> to.

If you're going to put jars in $SOLR_HOME/lib, then you should *only*
put jars in that directory, and NOT load jars explicitly.  The 
directives should not be used in solrconfig.xml when jars are loaded
from this directory, because Solr will automatically load jars from this
location and make them available to all cores.

If moving all your extra jars (including things like the dataimport jar)
to $SOLR_HOME/lib and taking out jar loading in solrconfig.xml doesn't
help, then depending on the Solr version, you *might* be running into
SOLR-6188.

https://issues.apache.org/jira/browse/SOLR-6188

You'll want to be sure that you don't the same jar more than once.  This
is the root of the specific problem that SOLR-6188 solves.  Loading the
same jar more than once can also happen if the jar is in the lib
directory AND mentioned on a  config element.

Thanks,
Shawn

How to Delta-Import to solr by Id(key word)

[fent]

I have a table with Id , this is a  increase attribute,
So I want to  Delta add new  category to solr may like select * from
my_table where Id  '${latest_id}'
the latest_id is the max Id that last time add ,
how to config the data-config.xml.
or how to get the max Id from the solr?

ths!






--
View this message in context: 
http://lucene.472066.n3.nabble.com/How-to-Delta-Import-to-solr-by-Id-key-word-tp4224090.html
Sent from the Solr - User mailing list archive at Nabble.com.

Import from Solr 4.10.4 into Solr 5.2.1 using DIH and SolrEntityProcessor

[Jens Ivar Jørdre]

Hi,

I wonder if someone may shed some light on an issue I have using DIH and 
SolrEntityProcessor. In short I am unable to import data from Solr 4.10.4 to 
5.2.1 where collection configurations in each have the very same schema.xml and 
solrconfig.xml. 

I have two Solr Cloud instances running. One cluster running Solr 4.10.4 has, 
at present, only one node. The other running Solr 5.2.1 has two nodes. I refer 
to the one running 4.10.4 as slr4 and one on the other cluster on which I 
operate below as slr5. Solr is running on port 8080 on slr4 and on port 8983 on 
slr5. Both slr4 and slr5 are CentOS boxes, release 6.6 and 7.1, respectively.

My DIH data-config.xml in slr5 is pretty much a copy of the one from 
https://wiki.apache.org/solr/DataImportHandler#SolrEntityProcessor and reads as 
follows:

?xml version=1.0 encoding=UTF-8 ?
dataConfig
document
entity name=sep processor=SolrEntityProcessor 
url=http://slr4:8080/solr/my_collection; fl=field1,field2,field3 wt=xml 
query=*:*/
/document
/dataConfig

I.e. I have added wt=xml as the import crosses major version boundary.

Now, running the data import from the web user interface of slr5 
(http://slr5:8983/solr/#/my_collection_shard1_replica1/dataimport//dataimport) 
my import times out!! My log output snippet below tells that connection to slr4 
by solrj was refused.

Caused by: java.lang.RuntimeException: 
org.apache.solr.handler.dataimport.DataImportHandlerException: 
org.apache.solr.client.solrj.SolrServerException: Server refused connection at: 
http://slr4:8080/solr/my_collection
at 
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:416)
at 
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:329)
at 
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:232)
... 29 more
Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException: 
org.apache.solr.client.solrj.SolrServerException: Server refused connection at: 
http://slr4:8080/solr/my_collection
at 
org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:62)
at 
org.apache.solr.handler.dataimport.SolrEntityProcessor.doQuery(SolrEntityProcessor.java:201)
at 
org.apache.solr.handler.dataimport.SolrEntityProcessor.buildIterator(SolrEntityProcessor.java:147)
at 
org.apache.solr.handler.dataimport.SolrEntityProcessor.nextRow(SolrEntityProcessor.java:128)
at 
org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:243)
at 
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:475)
at 
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:414)
... 31 more
Caused by: org.apache.solr.client.solrj.SolrServerException: Server refused 
connection at: http://slr4:8080/solr/my_collection
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:567)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:235)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:227)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:943)
at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:958)
at 
org.apache.solr.handler.dataimport.SolrEntityProcessor.doQuery(SolrEntityProcessor.java:198)
... 36 more
Caused by: java.net.ConnectException: Connection timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at 
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
at 
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at 
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at 
org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:117)
at 
org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
at 
org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
at 
org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
at 
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
at 
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107

Re: Import from Solr 4.10.4 into Solr 5.2.1 using DIH and SolrEntityProcessor

[Shawn Heisey]

On 7/10/2015 5:47 PM, Jens Ivar Jørdre wrote:
 I wonder if someone may shed some light on an issue I have using DIH and 
 SolrEntityProcessor. In short I am unable to import data from Solr 4.10.4 to 
 5.2.1 where collection configurations in each have the very same schema.xml 
 and solrconfig.xml. 

 I have two Solr Cloud instances running. One cluster running Solr 4.10.4 has, 
 at present, only one node. The other running Solr 5.2.1 has two nodes. I 
 refer to the one running 4.10.4 as slr4 and one on the other cluster on which 
 I operate below as slr5. Solr is running on port 8080 on slr4 and on port 
 8983 on slr5. Both slr4 and slr5 are CentOS boxes, release 6.6 and 7.1, 
 respectively.

 My DIH data-config.xml in slr5 is pretty much a copy of the one from 
 https://wiki.apache.org/solr/DataImportHandler#SolrEntityProcessor and reads 
 as follows:

 ?xml version=1.0 encoding=UTF-8 ?
 dataConfig
 document
 entity name=sep processor=SolrEntityProcessor 
 url=http://slr4:8080/solr/my_collection; fl=field1,field2,field3 wt=xml 
 query=*:*/
 /document
 /dataConfig

 I.e. I have added wt=xml as the import crosses major version boundary.

The javabin version changed exactly once -- from 1.4.1 to 3.1.0.  It has
not changed since then, so unless you are trying to import from a 1.x
version, you should not need xml.  Using xml will not hurt anything, but
javabin is more efficient.

 Now, running the data import from the web user interface of slr5 
 (http://slr5:8983/solr/#/my_collection_shard1_replica1/dataimport//dataimport)
  my import times out!! My log output snippet below tells that connection to 
 slr4 by solrj was refused.

 Caused by: java.lang.RuntimeException: 
 org.apache.solr.handler.dataimport.DataImportHandlerException: 
 org.apache.solr.client.solrj.SolrServerException: Server refused connection 
 at: http://slr4:8080/solr/my_collection

This sounds like a networking problem, which could be a firewall,
network misconfiguration on either host, or any number of other
problems.  The following later caused by message provides some
additional detail, but this is strange, because a timeout and a
connection refusal are very different failures:

 Caused by: java.net.ConnectException: Connection timed out

snip

 What does work, nonetheless, is to do curl 
 http://slr4:8080/solr/my_collection/select?q=*:* from slr5.

 Any one with some idea on how to solve this one?

I'm pretty sure that this is a problem that's more fundamental than
Solr.  It could be Java, the operating system, or something in your
network.  The fact that curl works but DIH doesn't is particularly strange.

Do you have any environment variables that might be configuring the
HttpClient in Solr to use a proxy server that either doesn't exist or is
not configured right?  That could explain what's going on.

Can you pull out any sensitive information and share your entire
solrconfig.xml and solr.xml files?

Thanks,
Shawn

Timestamp compatibility while performing delta import in solr

[archit2112]

Im new to solr.I have successfully indexed oracle 10g xe database. Im trying
to perform delta import on the same.
The Delta query required a comparison of last_modified column of the table
with ${dih.last_index_time}.
However in my application I do not have such a column . Also, i cannot add
this column. Therefore i used 'scn_to_timestamp(ora_rowscn)' to give the
value of the required timestamps. This query returns the value of type
timestamp in the following format 24-JUL-13 12.42.32.0 PM and
dih.last_index_time is in the format 2013-07-24 12:18:03. So, I changed the
format of dih.last_index_time as to_timestamp('${dih.last_index_time}',
'/MM/DD HH:MI:SS').

My Data-config looks like this - 

dataConfig
dataSource type=JdbcDataSource driver=oracle.jdbc.OracleDriver
url=jdbc:oracle:thin:@160.110.13.49:1521:xe user=system
password=manager / 
document name=product_info
entity name=PRODUCT pk=PID query=SELECT * FROM PRODUCT
deltaImportQuery=SELECT * FROM PRODUCT WHERE PID=${dih.delta.id}
deltaQuery=SELECT PID FROM PRODUCT WHERE scn_to_timestamp(ora_rowscn) 
to_timestamp('${dih.last_index_time}', '/MM/DD HH:MI:SS')
  field column=PID name=id / 
  field column=PNAME name=itemName / 
  field column=INITQTY name=itemQuantity /
  field column=REMQTY name=remQuantity /
  field column=PRICE name=itemPrice / 
  field column=SPECIFICATION name=specifications / 
  entity name=SUB_CATEGORY query=SELECT * FROM SUB_CATEGORY WHERE
SCID=${PRODUCT.SCID}
field column=SUBCATNAME name=brand / 
entity name=CATEGORY query=SELECT CNAME FROM CATEGORY WHERE
CID=${SUB_CATEGORY.CID}
  field column=CNAME name=itemCategory / 
/entity
  /entity
/entity
/document
/dataConfig

However,This is not working and im getting the following error - 
Unable to execute query: SELECT * FROM PRODUCT WHERE PID= Processing
Document # 1
Caused by: java.sql.SQLException: ORA-00936: missing expression

Please help me out!!!



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Timestamp-compatibility-while-performing-delta-import-in-solr-tp4079982.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Timestamp compatibility while performing delta import in solr

[Shawn Heisey]

On 7/24/2013 3:32 AM, archit2112 wrote:
 However,This is not working and im getting the following error - 
 Unable to execute query: SELECT * FROM PRODUCT WHERE PID= Processing
 Document # 1
 Caused by: java.sql.SQLException: ORA-00936: missing expression

Here's your first entity:

entity name=PRODUCT pk=PID query=SELECT * FROM PRODUCT
deltaImportQuery=SELECT * FROM PRODUCT WHERE PID=${dih.delta.id}
deltaQuery=SELECT PID FROM PRODUCT WHERE scn_to_timestamp(ora_rowscn) 
to_timestamp('${dih.last_index_time}', '/MM/DD HH:MI:SS')

It seems that when your deltaQuery executes, the first document has no
value for PID.  If PID is indeed the primary key for the 'PRODUCT'
table, which is what you have told DIH is the case, then the only thing
that makes sense is that it is a blank value, rather than NULL.

If you have something else in your database that is a better indicator
of what's new than a timestamp, you can use that, you just have to pass
it in as a parameter when you access the dataimport URL by HTTP.  If the
parameter on the URL is mycolumn=myvalue then you can use
${dih.request.mycolumn} in your config file.

Thanks,
Shawn

Re: Import in Solr

[William Bell]

You can update a row. Just allow a request parameter in the DIH and add it
to your query.

id=65

Then in your query you can use that. See the Wiki on DIh.



On Friday, April 19, 2013, Gora Mohanty wrote:

 On 19 April 2013 19:50, hassancrowdc hassancrowdc...@gmail.comjavascript:;
 wrote:
  I want to update(delta-import) one specific item. Is there any query to
 do
  that?

 No.

 Regards,
 Gora



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Import in Solr

[hassancrowdc]

I want to update(delta-import) one specific item. Is there any query to do
that? 

like i can delete specific item with the following query: 

localhost:8080/solr/devices/update?stream.body=deletequeryid:46/query/deletecommit=true

Thanks.



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Import-in-Solr-tp4057301.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Import in Solr

[Gora Mohanty]

On 19 April 2013 19:50, hassancrowdc hassancrowdc...@gmail.com wrote:
 I want to update(delta-import) one specific item. Is there any query to do
 that?

No.

Regards,
Gora

Re: Large import making solr unresponsive

[Otis Gospodnetic]

Hi Brent,

You said from what I can tell there is no disk, network, or memory pressure 
- maybe you can share what and how you checked this? (see my signature for
a tool that can help with this)

I'm asking because the above is in conflict with responses from solr still
come back with a 10ms qtime, which indicate search itself was fast, but
either disk or network were slow.  Try with rows=big number here and
rows=0 and that will give you an idea where to look.

Otis
--
SOLR Performance Monitoring - http://sematext.com/spm/index.html





On Mon, Dec 17, 2012 at 1:04 AM, Brent Mills bmi...@uship.com wrote:

 This is an issue we've only been running into lately so I'm not sure what
 to make of it.  We have 2 cores on a solr machine right now, one of them is
 about 10k documents, the other is about 1.5mil.  None of the documents are
 very large, only about 30 short attributes.  We also have about 10
 requests/sec hitting the smaller core and less on the larger one.  Whenever
 we try to do a full import on the smaller one everything is fine, the
 response times stay the same during the whole 30 seconds it takes to run
 the indexer.  The cpu also stays fairly low.

 When we run a full import on the larger one the response times on all
 cores tank from about 10ms to over 8 seconds.  We have a 4 core machine
 (VM) and I've noticed 1 core stays pegged the entire time which is
 understandable since the DIH as I understand it is single threaded.  Also,
 from what I can tell there is no disk, network, or memory pressure (8gb)
 either and the other procs do virtually nothing.  Also the responses from
 solr still come back with a 10ms qtime.  My best guess at this point is
 tomcat is having issues when the single proc gets pegged but I'm at a loss
 on how to further diagnose this to a tomcat issue or something weird that
 solr is doing.

 Has anyone run into this before or have ideas about what might be
 happening?

Large import making solr unresponsive

[Brent Mills]

This is an issue we've only been running into lately so I'm not sure what to 
make of it.  We have 2 cores on a solr machine right now, one of them is about 
10k documents, the other is about 1.5mil.  None of the documents are very 
large, only about 30 short attributes.  We also have about 10 requests/sec 
hitting the smaller core and less on the larger one.  Whenever we try to do a 
full import on the smaller one everything is fine, the response times stay the 
same during the whole 30 seconds it takes to run the indexer.  The cpu also 
stays fairly low.

When we run a full import on the larger one the response times on all cores 
tank from about 10ms to over 8 seconds.  We have a 4 core machine (VM) and I've 
noticed 1 core stays pegged the entire time which is understandable since the 
DIH as I understand it is single threaded.  Also, from what I can tell there is 
no disk, network, or memory pressure (8gb) either and the other procs do 
virtually nothing.  Also the responses from solr still come back with a 10ms 
qtime.  My best guess at this point is tomcat is having issues when the single 
proc gets pegged but I'm at a loss on how to further diagnose this to a tomcat 
issue or something weird that solr is doing.

Has anyone run into this before or have ideas about what might be happening?