Re: Migrating from Solr 6.X to Solr 7.X: "non legacy mode coreNodeName missing"

[Erick Erickson]

If at all possible you shouldn't run with legacyCloud=true. I was just
mentioning that in order to see if that was the place to start
looking, I could have been clearer.

Let's back up a bit though. I just tried this creating a collection
with 6.6.0, changing to 7.1.0 and I can add replicas just fine.

Things are weirder when creating a collection with 6.6.1 (or 7.0 for
that matter), adding a new replica works but the _old_ core fails to
load.

So what _exactly_ are your versions? And what _exactly_ did you try?
This is suspiciously like
https://issues.apache.org/jira/browse/SOLR-11503, the fix will be in
7.2 or you can try patching your 7.1...

Erick

On Mon, Oct 30, 2017 at 1:00 PM, Dave Seltzer  wrote:
> Thanks Erick, I've looked over the documentation.
>
> Quick follow-up question:
>
> What are the consequences of running with legacyCloud=true?
> Would I need to point a new Solr cluster at a new Zookeeper instance to
> avoid this?
>
> Many thanks!
>
> -Dave
>
> On Mon, Oct 30, 2017 at 1:36 PM, Erick Erickson 
> wrote:
>
>> empty clusterstate.json node. OK, you're on state format 2 (the modern
>> one) so that's good. No need to MIGRATESTATE
>>
>> The admin UI>>cloud>>tree view should have a cluster.properties (IIRC
>> the name correctly, it'll be fairly obvious). If you have no
>> non-default values it'll be totally missing. It's a sibling znode to
>> clusterstate.json.
>>
>> On page 56 of the 7.0 manual you'll find the command to set
>> legacyCloud. I recommend you download the PDF as it's easier to search
>> until someone breaks loose the time to get the new doc site search
>> working
>>
>> Here's the short form:
>>
>> ./server/scripts/cloud-scripts/zkcli.sh -zkhost 127.0.0.1:2181 -cmd
>> clusterprop -name legacyCloud -val true
>>
>> If that works you'll see a node in your ZK tree.
>>
>> Best,
>> Erick
>>
>> On Mon, Oct 30, 2017 at 9:13 AM, Dave Seltzer  wrote:
>> >> You may have to set legacyCloud=true in your cluster properties
>> >> Check your cluster properties (top level znode in ZooKeeper).
>> >
>> > How would I check this? Would it appear in the Solr Cloud interface, or
>> do
>> > I have to query Zookeeper?
>> >
>> >> Or, try the MIGRATESTATEFORMAT collection API on your 6x collection
>> >> first. If your current installation has a single clusterstate.json
>> >> that contains all of your collection information, then this would be
>> >> indicated.
>> >
>> > There is a single /clusterstate.json but it is an empty object. Rather,
>> > each collection has a state.json which seems to indicate the state of the
>> > cluster.
>> >
>> > Does that mean I'm using the "zookeeper is the truth" system or the old
>> > system?
>> >
>> > Thanks!
>> >
>> > -Dave
>> >
>> >
>> > On Mon, Oct 30, 2017 at 11:55 AM, Erick Erickson <
>> erickerick...@gmail.com>
>> > wrote:
>> >
>> >> You may have to set legacyCloud=true in your cluster properties. In
>> >> the Solr reference guide (7.0) there's a section "Changes to Default
>> >> Behaviors" that'll tell you a bit about that. Check your cluster
>> >> properties (top level znode in ZooKeeper).
>> >>
>> >> Or, try the MIGRATESTATEFORMAT collection API on your 6x collection
>> >> first. If your current installation has a single clusterstate.json
>> >> that contains all of your collection information, then this would be
>> >> indicated.
>> >>
>> >> Mind you I'd experiment on a non-prod system first
>> >>
>> >> Best,
>> >> Erick
>> >>
>> >>
>> >>
>> >> On Mon, Oct 30, 2017 at 8:44 AM, Dave Seltzer 
>> wrote:
>> >> > I have a Solr collection with 1 shard and around 40 replicas hosted on
>> >> Solr
>> >> > 6.6.0
>> >> >
>> >> > I added a Solr 7.1 server to the cluster and then tried to add a
>> replica
>> >> > (ADDREPLICA) onto the new server but got this message:
>> >> >
>> >> >> Error CREATEing SolrCore 'content_collection_20171013_
>> >> shard1_replica37':
>> >> > non legacy mode coreNodeName missing {collection.configName=content,
>> >> > shard=shard1, collection=content_collection_20171013}
>> >> >
>> >> > Is there something I have to do to prepare this collection for Solr
>> 7.x?
>> >> >
>> >> > Thanks,
>> >> >
>> >> > -Dave
>> >> >
>> >> > [root@crompcoreph02 ~]# curl "http://
>> >> > [solrclusterloadbalancer]/solr/admin/collections?action=
>> >> ADDREPLICA=content_collection_20171013=
>> >> shard1=$crompcoreph02:8983_solr"
>> >> > 
>> >> > 
>> >> > 500> >> > name="QTime">222> >> > name="crompcoreph02:8983_solr">org.apache.solr.client.solrj.
>> >> impl.HttpSolrClient$RemoteSolrException:Error
>> >> > from server at http://crompcoreph02:8983/solr: Error CREATEing
>> SolrCore
>> >> > 'content_collection_20171013_shard1_replica37': non legacy mode
>> >> > coreNodeName missing {collection.configName=content, shard=shard1,
>> >> > collection=content_collection_20171013}> name="Operation
>> >> > addreplica caused
>> >> > exception:">org.apache.solr.common.SolrException:org.
>> 

Re: jetty access logs for monitoring

[Otis Gospodnetić]

Hi,

I think Logagent  has support for Solr
logs.  Aha, yes, it does:
https://github.com/sematext/logagent-js/blob/master/patterns.yml (search
for Solr).

But if you are looking for Solr metrics monitoring, see Solr monitoring
 in Sematext Cloud
(disclosure: see signature :)).

HTH

Otis
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/


On Mon, Oct 30, 2017 at 3:56 PM, j.s.  wrote:

> hi
>
> my current problem is monitoring my solr instance. it would be best if i
> could get a jetty access log to appear. i think i can do this with the log4
> interface, but i have not yet figured out the appropriate configuration
> changes.
>
> can someone direct me toward some information on this? i'm running solr
> 6.6. i've looked in the ref guide and around the wiki and have not found
> anything that answers my question. the stuff i've found is for solr 4.X ...
>
> any help is welcome, thx.
>

Re: Failed to create collection SOLR 6.3 HDP 2.6.2

[Dan Caulfield]

Shawn Heisey-2 wrote
> On 10/26/2017 10:29 AM, Dan Caulfield wrote:
>> I'm creating a collection on a new cluster.  There are six new Solr nodes
>> using a HDP 2.6.2 cluster for storage.  Has anyone seen similar errors?
> 
>> ERROR: Failed to create collection 'maxis_clickstream' due to:
>> {10.126.191.24:8983_solr=org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error
>> from server at http://10.126.191.24:8983/solr: Error CREATEing SolrCore
>> 'maxis_clickstream_shard6_replica1': Unable to create core
>> [maxis_clickstream_shard6_replica1] Caused by: no segments* file found in
>> LockValidatingDirectoryWrapper(NRTCachingDirectory(BlockDirectory(HdfsDirectory@hdfs://edwbitstmil/apps/solr/data/maxis_clickstream/core_node4/data/index
> 
> This error looks like what happens when the index directory already
> exists but contains no files. This specific error log mentions the
> following location:
> 
> hdfs://edwbitstmil/apps/solr/data/maxis_clickstream/core_node4/data/index
> 
> If I'm right, then you'll need to rename or delete that index directory
> on your HDFS store in order for the core creation to work.  The log
> snippet you shared included this error on several index directories, so
> you will need to repeat the rename or delete on all of them that show
> the error.
> 
> If you can be absolutely certain that there is no data contained there
> that is actively being used, then I *think* you could take care of
> everything in one step by renaming the maxis_clickstream directory to
> something else so that Solr will recreate the entire directory
> structure.  I am not recommending *deleting* that directory, because I
> have no idea whether it might contain something you'll want to keep.
> 
> The underlying problem I'm describing happens at the Lucene layer, and
> Solr is unable to do anything about it.  In order for Lucene to create a
> new index, the index directory must not be there at all.  Deleting the
> contents of the index directory but leaving the directory will cause
> errors the next time Lucene tries to use that directory.
> 
> Thanks,
> Shawn

Thanks Shawn, 
I tried your recommended solution and delete the maxis_clickstream
directory.  I got the same error when trying to recreate the collection. 
Can you think of anything else to try?  




--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Migrating from Solr 6.X to Solr 7.X: "non legacy mode coreNodeName missing"

[Dave Seltzer]

Thanks Erick, I've looked over the documentation.

Quick follow-up question:

What are the consequences of running with legacyCloud=true?
Would I need to point a new Solr cluster at a new Zookeeper instance to
avoid this?

Many thanks!

-Dave

On Mon, Oct 30, 2017 at 1:36 PM, Erick Erickson 
wrote:

> empty clusterstate.json node. OK, you're on state format 2 (the modern
> one) so that's good. No need to MIGRATESTATE
>
> The admin UI>>cloud>>tree view should have a cluster.properties (IIRC
> the name correctly, it'll be fairly obvious). If you have no
> non-default values it'll be totally missing. It's a sibling znode to
> clusterstate.json.
>
> On page 56 of the 7.0 manual you'll find the command to set
> legacyCloud. I recommend you download the PDF as it's easier to search
> until someone breaks loose the time to get the new doc site search
> working
>
> Here's the short form:
>
> ./server/scripts/cloud-scripts/zkcli.sh -zkhost 127.0.0.1:2181 -cmd
> clusterprop -name legacyCloud -val true
>
> If that works you'll see a node in your ZK tree.
>
> Best,
> Erick
>
> On Mon, Oct 30, 2017 at 9:13 AM, Dave Seltzer  wrote:
> >> You may have to set legacyCloud=true in your cluster properties
> >> Check your cluster properties (top level znode in ZooKeeper).
> >
> > How would I check this? Would it appear in the Solr Cloud interface, or
> do
> > I have to query Zookeeper?
> >
> >> Or, try the MIGRATESTATEFORMAT collection API on your 6x collection
> >> first. If your current installation has a single clusterstate.json
> >> that contains all of your collection information, then this would be
> >> indicated.
> >
> > There is a single /clusterstate.json but it is an empty object. Rather,
> > each collection has a state.json which seems to indicate the state of the
> > cluster.
> >
> > Does that mean I'm using the "zookeeper is the truth" system or the old
> > system?
> >
> > Thanks!
> >
> > -Dave
> >
> >
> > On Mon, Oct 30, 2017 at 11:55 AM, Erick Erickson <
> erickerick...@gmail.com>
> > wrote:
> >
> >> You may have to set legacyCloud=true in your cluster properties. In
> >> the Solr reference guide (7.0) there's a section "Changes to Default
> >> Behaviors" that'll tell you a bit about that. Check your cluster
> >> properties (top level znode in ZooKeeper).
> >>
> >> Or, try the MIGRATESTATEFORMAT collection API on your 6x collection
> >> first. If your current installation has a single clusterstate.json
> >> that contains all of your collection information, then this would be
> >> indicated.
> >>
> >> Mind you I'd experiment on a non-prod system first
> >>
> >> Best,
> >> Erick
> >>
> >>
> >>
> >> On Mon, Oct 30, 2017 at 8:44 AM, Dave Seltzer 
> wrote:
> >> > I have a Solr collection with 1 shard and around 40 replicas hosted on
> >> Solr
> >> > 6.6.0
> >> >
> >> > I added a Solr 7.1 server to the cluster and then tried to add a
> replica
> >> > (ADDREPLICA) onto the new server but got this message:
> >> >
> >> >> Error CREATEing SolrCore 'content_collection_20171013_
> >> shard1_replica37':
> >> > non legacy mode coreNodeName missing {collection.configName=content,
> >> > shard=shard1, collection=content_collection_20171013}
> >> >
> >> > Is there something I have to do to prepare this collection for Solr
> 7.x?
> >> >
> >> > Thanks,
> >> >
> >> > -Dave
> >> >
> >> > [root@crompcoreph02 ~]# curl "http://
> >> > [solrclusterloadbalancer]/solr/admin/collections?action=
> >> ADDREPLICA=content_collection_20171013=
> >> shard1=$crompcoreph02:8983_solr"
> >> > 
> >> > 
> >> > 500 >> > name="QTime">222 >> > name="crompcoreph02:8983_solr">org.apache.solr.client.solrj.
> >> impl.HttpSolrClient$RemoteSolrException:Error
> >> > from server at http://crompcoreph02:8983/solr: Error CREATEing
> SolrCore
> >> > 'content_collection_20171013_shard1_replica37': non legacy mode
> >> > coreNodeName missing {collection.configName=content, shard=shard1,
> >> > collection=content_collection_20171013} name="Operation
> >> > addreplica caused
> >> > exception:">org.apache.solr.common.SolrException:org.
> apache.solr.common.
> >> SolrException:
> >> > ADDREPLICA failed to create replica >> > name="msg">ADDREPLICA failed to create replica >> > name="rspCode">500 >> name="metadata"> >> > name="error-class">org.apache.solr.common.SolrException >> > name="root-error-class">org.apache.solr.common.
> >> SolrException >> > name="msg">ADDREPLICA failed to create replica >> > name="trace">org.apache.solr.common.SolrException: ADDREPLICA failed
> to
> >> > create replica
> >> > at
> >> > org.apache.solr.handler.admin.CollectionsHandler.handleResponse(
> >> CollectionsHandler.java:300)
> >> > at
> >> > org.apache.solr.handler.admin.CollectionsHandler.invokeAction(
> >> CollectionsHandler.java:237)
> >> > at
> >> > org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(
> >> CollectionsHandler.java:215)
> >> > at
> >> > 

jetty access logs for monitoring

[j.s.]

hi

my current problem is monitoring my solr instance. it would be best if i 
could get a jetty access log to appear. i think i can do this with the 
log4 interface, but i have not yet figured out the appropriate 
configuration changes.


can someone direct me toward some information on this? i'm running solr 
6.6. i've looked in the ref guide and around the wiki and have not found 
anything that answers my question. the stuff i've found is for solr 4.X ...


any help is welcome, thx.

Replica node down NullPointerException

[Webster Homer]

I have a Replica marked as down in Production, but the diagnostics as to
why it's down are useless. All we see is a NullPointerException

I see this error message in the log:
2017-10-30 14:17:39.008 ERROR (qtp472654579-39773) [ ] o.a.s.s.HttpSolrCall
null:org.apache.solr.common.SolrException: SolrCore
'bb-catalog-product_shard1_replica2' is not available due to init failure:
null


This is the stack trace:
Exception in thread "Thread-7970" org.apache.solr.common.SolrException:
SolrCore 'bb-catalog-product_shard1_replica2' is not available due to init
failure: null
at org.apache.solr.core.CoreContainer.getCore(
CoreContainer.java:1092)
at org.apache.solr.handler.admin.CoreAdminOperation.lambda$
null$9(CoreAdminOperation.java:566)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at org.apache.solr.update.TransactionLog$FSReverseReader.(
TransactionLog.java:715)
at org.apache.solr.update.TransactionLog.getReverseReader(
TransactionLog.java:597)
at org.apache.solr.update.UpdateLog$RecentUpdates.
update(UpdateLog.java:995)
at org.apache.solr.update.UpdateLog$RecentUpdates.
(UpdateLog.java:935)
at org.apache.solr.update.UpdateLog.getRecentUpdates(
UpdateLog.java:1099)
at org.apache.solr.update.UpdateLog.seedBucketsWithHighestVersion(
UpdateLog.java:1575)
at org.apache.solr.update.UpdateLog.seedBucketsWithHighestVersion(
UpdateLog.java:1603)
at org.apache.solr.core.SolrCore.seedVersionBuckets(SolrCore.
java:859)
at org.apache.solr.core.SolrCore.(SolrCore.java:843)
at org.apache.solr.core.SolrCore.(SolrCore.java:688)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:838)
at org.apache.solr.core.CoreContainer.lambda$load$0(
CoreContainer.java:494)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$
MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:624)
... 1 more
Exception in thread "Thread-7971" org.apache.solr.common.SolrException:
SolrCore 'bb-catalog-product_shard1_replica2' is not available due to init
failure: null
at org.apache.solr.core.CoreContainer.getCore(
CoreContainer.java:1092)
at org.apache.solr.handler.admin.CoreAdminOperation.lambda$
null$9(CoreAdminOperation.java:566)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at org.apache.solr.update.TransactionLog$FSReverseReader.(
TransactionLog.java:715)
at org.apache.solr.update.TransactionLog.getReverseReader(
TransactionLog.java:597)
at org.apache.solr.update.UpdateLog$RecentUpdates.
update(UpdateLog.java:995)
at org.apache.solr.update.UpdateLog$RecentUpdates.
(UpdateLog.java:935)
at org.apache.solr.update.UpdateLog.getRecentUpdates(
UpdateLog.java:1099)
at org.apache.solr.update.UpdateLog.seedBucketsWithHighestVersion(
UpdateLog.java:1575)
at org.apache.solr.update.UpdateLog.seedBucketsWithHighestVersion(
UpdateLog.java:1603)
at org.apache.solr.core.SolrCore.seedVersionBuckets(SolrCore.
java:859)
at org.apache.solr.core.SolrCore.(SolrCore.java:843)
at org.apache.solr.core.SolrCore.(SolrCore.java:688)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:838)
at org.apache.solr.core.CoreContainer.lambda$load$0(
CoreContainer.java:494)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$
MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:624)
... 1 more

This is from a Solr 6.2.0 solrcloud

This is a cdcr replication target

There was an error in a Zookeeper log, it's hard to know if it is symptom
or a cause
2017-10-28 14:42:18,951 [myid:2] - ERROR [LearnerHandler-/10.121.0.61:43192
:LearnerHandler@631] - Unexpected exception causing shutdown while sock
still open
java.net.SocketException: Connection reset
   at java.net.SocketInputStream.read(SocketInputStream.java:210)
   at java.net.SocketInputStream.read(SocketInputStream.java:141)
   at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
   at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
   at java.io.DataInputStream.readInt(DataInputStream.java:387)
   at
org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
   at
org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
   at

Re: Migrating from Solr 6.X to Solr 7.X: "non legacy mode coreNodeName missing"

[Erick Erickson]

empty clusterstate.json node. OK, you're on state format 2 (the modern
one) so that's good. No need to MIGRATESTATE

The admin UI>>cloud>>tree view should have a cluster.properties (IIRC
the name correctly, it'll be fairly obvious). If you have no
non-default values it'll be totally missing. It's a sibling znode to
clusterstate.json.

On page 56 of the 7.0 manual you'll find the command to set
legacyCloud. I recommend you download the PDF as it's easier to search
until someone breaks loose the time to get the new doc site search
working

Here's the short form:

./server/scripts/cloud-scripts/zkcli.sh -zkhost 127.0.0.1:2181 -cmd
clusterprop -name legacyCloud -val true

If that works you'll see a node in your ZK tree.

Best,
Erick

On Mon, Oct 30, 2017 at 9:13 AM, Dave Seltzer  wrote:
>> You may have to set legacyCloud=true in your cluster properties
>> Check your cluster properties (top level znode in ZooKeeper).
>
> How would I check this? Would it appear in the Solr Cloud interface, or do
> I have to query Zookeeper?
>
>> Or, try the MIGRATESTATEFORMAT collection API on your 6x collection
>> first. If your current installation has a single clusterstate.json
>> that contains all of your collection information, then this would be
>> indicated.
>
> There is a single /clusterstate.json but it is an empty object. Rather,
> each collection has a state.json which seems to indicate the state of the
> cluster.
>
> Does that mean I'm using the "zookeeper is the truth" system or the old
> system?
>
> Thanks!
>
> -Dave
>
>
> On Mon, Oct 30, 2017 at 11:55 AM, Erick Erickson 
> wrote:
>
>> You may have to set legacyCloud=true in your cluster properties. In
>> the Solr reference guide (7.0) there's a section "Changes to Default
>> Behaviors" that'll tell you a bit about that. Check your cluster
>> properties (top level znode in ZooKeeper).
>>
>> Or, try the MIGRATESTATEFORMAT collection API on your 6x collection
>> first. If your current installation has a single clusterstate.json
>> that contains all of your collection information, then this would be
>> indicated.
>>
>> Mind you I'd experiment on a non-prod system first
>>
>> Best,
>> Erick
>>
>>
>>
>> On Mon, Oct 30, 2017 at 8:44 AM, Dave Seltzer  wrote:
>> > I have a Solr collection with 1 shard and around 40 replicas hosted on
>> Solr
>> > 6.6.0
>> >
>> > I added a Solr 7.1 server to the cluster and then tried to add a replica
>> > (ADDREPLICA) onto the new server but got this message:
>> >
>> >> Error CREATEing SolrCore 'content_collection_20171013_
>> shard1_replica37':
>> > non legacy mode coreNodeName missing {collection.configName=content,
>> > shard=shard1, collection=content_collection_20171013}
>> >
>> > Is there something I have to do to prepare this collection for Solr 7.x?
>> >
>> > Thanks,
>> >
>> > -Dave
>> >
>> > [root@crompcoreph02 ~]# curl "http://
>> > [solrclusterloadbalancer]/solr/admin/collections?action=
>> ADDREPLICA=content_collection_20171013=
>> shard1=$crompcoreph02:8983_solr"
>> > 
>> > 
>> > 500> > name="QTime">222> > name="crompcoreph02:8983_solr">org.apache.solr.client.solrj.
>> impl.HttpSolrClient$RemoteSolrException:Error
>> > from server at http://crompcoreph02:8983/solr: Error CREATEing SolrCore
>> > 'content_collection_20171013_shard1_replica37': non legacy mode
>> > coreNodeName missing {collection.configName=content, shard=shard1,
>> > collection=content_collection_20171013}org.apache.solr.common.SolrException:org.apache.solr.common.
>> SolrException:
>> > ADDREPLICA failed to create replica> > name="msg">ADDREPLICA failed to create replica> > name="rspCode">500> name="metadata">> > name="error-class">org.apache.solr.common.SolrException> > name="root-error-class">org.apache.solr.common.
>> SolrException> > name="msg">ADDREPLICA failed to create replica> > name="trace">org.apache.solr.common.SolrException: ADDREPLICA failed to
>> > create replica
>> > at
>> > org.apache.solr.handler.admin.CollectionsHandler.handleResponse(
>> CollectionsHandler.java:300)
>> > at
>> > org.apache.solr.handler.admin.CollectionsHandler.invokeAction(
>> CollectionsHandler.java:237)
>> > at
>> > org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(
>> CollectionsHandler.java:215)
>> > at
>> > org.apache.solr.handler.RequestHandlerBase.handleRequest(
>> RequestHandlerBase.java:173)
>> > at
>> > org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:748)
>> > at
>> > org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(
>> HttpSolrCall.java:729)
>> > at org.apache.solr.servlet.HttpSolrCall.call(
>> HttpSolrCall.java:510)
>> > at
>> > org.apache.solr.servlet.SolrDispatchFilter.doFilter(
>> SolrDispatchFilter.java:361)
>> > at
>> > org.apache.solr.servlet.SolrDispatchFilter.doFilter(
>> SolrDispatchFilter.java:305)
>> > at
>> > 

RE: LTR feature extraction performance issues

[Brian Yee]

I'm still having this issue. Does anyone have LTR feature extraction 
successfully running and have cache inserts/hits?

--Brian

-Original Message-
From: Brian Yee [mailto:b...@wayfair.com] 
Sent: Tuesday, October 24, 2017 12:14 PM
To: solr-user@lucene.apache.org
Subject: RE: LTR feature extraction performance issues

Hi Alessandro,

Unfortunately some of my most important features are query dependent. I think I 
found an issue though. I don't think my features are being inserted into the 
cache. Notice "cumulative_inserts:0". There are a lot of lookups, but since 
there appear to be no values in the cache, the hitratio is 0.

stats:
cumulative_evictions:0
cumulative_hitratio:0
cumulative_hits:0
cumulative_inserts:0
cumulative_lookups:215319
evictions:0
hitratio:0
hits:0
inserts:0
lookups:3303
size:0
warmupTime:0


My configs look are as follows:



  

  
QUERY_DOC_FV
sparse
  

Would anyone have any idea why my features are not being inserted into the 
cache? Is there an additional config setting I need?


--Brian

-Original Message-
From: alessandro.benedetti [mailto:a.benede...@sease.io] 
Sent: Monday, October 23, 2017 10:01 AM
To: solr-user@lucene.apache.org
Subject: Re: LTR feature extraction performance issues

It strictly depends on the kind of features you are using.
At the moment there is just one cache for all the features.
This means that even if you have 1 query dependent feature and 100 document 
dependent feature, a different value for the query dependent one will 
invalidate the cache entry for the full vector[1].

You may look to optimise your features ( where possible).

[1]  https://issues.apache.org/jira/browse/SOLR-10448



-
---
Alessandro Benedetti
Search Consultant, R Software Engineer, Director Sease Ltd. - www.sease.io
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: mvn test failing

[Steve Rowe]

Hi Tariq,

It’s difficult to tell what happened without seeing the logs from the failed 
test(s).  (The commands you issued look fine.)

--
Steve
www.lucidworks.com

> On Oct 29, 2017, at 1:48 AM, Tarique Anwer  wrote:
> 
> hi,
> 
> I am new to Solr.
> I am trying to build Solr from source code using Maven.
> So I performed the following steps:
> 
> 1. Download the source code zip from https://github.com/apache/lucene-solr
> 2. unzip & run from top level dir:
>  $ ant get-maven-poms
> $ cd maven-build
> 
> 3. then build:
>  $ mvn -DskipTests install
> 
> Which shows that build is successful.
> 
> So I tried to run the tests afterwords:
>  $ mvn test
> 
> But tests are failing:
> 
> [INFO] Apache Solr Analysis Extras  FAILURE [02:48
> min]
> [INFO] Apache Solr Core tests . SKIPPED
> [INFO] Apache Solr Core aggregator POM  SKIPPED
> [INFO] Apache Solr Solrj tests  SKIPPED
> [INFO] Apache Solr Solrj aggregator POM ... SKIPPED
> [INFO] Apache Solr Analytics Package .. SKIPPED
> [INFO] Apache Solr Clustering . SKIPPED
> [INFO] Apache Solr DataImportHandler .. SKIPPED
> [INFO] Apache Solr DataImportHandler Extras ... SKIPPED
> [INFO] Apache Solr Content Extraction Library . SKIPPED
> [INFO] Apache Solr Language Identifier  SKIPPED
> [INFO] Apache Solr Learning to Rank Package ... SKIPPED
> [INFO] Apache Solr UIMA integration ... SKIPPED
> [INFO] Apache Solr Velocity ... SKIPPED
> [INFO] Apache Solr Contrib aggregator POM . SKIPPED
> [INFO]
> 
> [INFO] BUILD FAILURE
> [INFO]
> 
> [INFO] Total time: 17:45 min
> [INFO] Finished at: 2017-10-29T05:46:43Z
> [INFO] Final Memory: 194M/1999M
> [INFO]
> 
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on
> project solr-analysis-extras: There are test failures.
> [ERROR]
> [ERROR] Please refer to
> /home/ec2-user/tariq/lucene-solr-master/maven-build/solr/contrib/analysis-extras/target/surefire-reports
> for the individual test results.
> [ERROR] -> [Help 1]
> 
> 
> 
> Did I do something wrong? Or i missed some steps before build.
> Any help is highly appreciated.
> 
> 
> ​With Regards,​
> 
> Tariq

[Parent] doc transformer

[Aurélien MAZOYER]

Hi,

 

Is there in Solr a kind of [parent] doc transformer (like the [child] doc
transformer) that can be used to embed parent’s fields in the response of a
query that uses the block join children query parser?

 

Thank you,

 

Aurélien MAZOYER

Re: Migrating from Solr 6.X to Solr 7.X: "non legacy mode coreNodeName missing"

[Dave Seltzer]

> You may have to set legacyCloud=true in your cluster properties
> Check your cluster properties (top level znode in ZooKeeper).

How would I check this? Would it appear in the Solr Cloud interface, or do
I have to query Zookeeper?

> Or, try the MIGRATESTATEFORMAT collection API on your 6x collection
> first. If your current installation has a single clusterstate.json
> that contains all of your collection information, then this would be
> indicated.

There is a single /clusterstate.json but it is an empty object. Rather,
each collection has a state.json which seems to indicate the state of the
cluster.

Does that mean I'm using the "zookeeper is the truth" system or the old
system?

Thanks!

-Dave


On Mon, Oct 30, 2017 at 11:55 AM, Erick Erickson 
wrote:

> You may have to set legacyCloud=true in your cluster properties. In
> the Solr reference guide (7.0) there's a section "Changes to Default
> Behaviors" that'll tell you a bit about that. Check your cluster
> properties (top level znode in ZooKeeper).
>
> Or, try the MIGRATESTATEFORMAT collection API on your 6x collection
> first. If your current installation has a single clusterstate.json
> that contains all of your collection information, then this would be
> indicated.
>
> Mind you I'd experiment on a non-prod system first
>
> Best,
> Erick
>
>
>
> On Mon, Oct 30, 2017 at 8:44 AM, Dave Seltzer  wrote:
> > I have a Solr collection with 1 shard and around 40 replicas hosted on
> Solr
> > 6.6.0
> >
> > I added a Solr 7.1 server to the cluster and then tried to add a replica
> > (ADDREPLICA) onto the new server but got this message:
> >
> >> Error CREATEing SolrCore 'content_collection_20171013_
> shard1_replica37':
> > non legacy mode coreNodeName missing {collection.configName=content,
> > shard=shard1, collection=content_collection_20171013}
> >
> > Is there something I have to do to prepare this collection for Solr 7.x?
> >
> > Thanks,
> >
> > -Dave
> >
> > [root@crompcoreph02 ~]# curl "http://
> > [solrclusterloadbalancer]/solr/admin/collections?action=
> ADDREPLICA=content_collection_20171013=
> shard1=$crompcoreph02:8983_solr"
> > 
> > 
> > 500 > name="QTime">222 > name="crompcoreph02:8983_solr">org.apache.solr.client.solrj.
> impl.HttpSolrClient$RemoteSolrException:Error
> > from server at http://crompcoreph02:8983/solr: Error CREATEing SolrCore
> > 'content_collection_20171013_shard1_replica37': non legacy mode
> > coreNodeName missing {collection.configName=content, shard=shard1,
> > collection=content_collection_20171013}org.apache.solr.common.SolrException:org.apache.solr.common.
> SolrException:
> > ADDREPLICA failed to create replica > name="msg">ADDREPLICA failed to create replica > name="rspCode">500 name="metadata"> > name="error-class">org.apache.solr.common.SolrException > name="root-error-class">org.apache.solr.common.
> SolrException > name="msg">ADDREPLICA failed to create replica > name="trace">org.apache.solr.common.SolrException: ADDREPLICA failed to
> > create replica
> > at
> > org.apache.solr.handler.admin.CollectionsHandler.handleResponse(
> CollectionsHandler.java:300)
> > at
> > org.apache.solr.handler.admin.CollectionsHandler.invokeAction(
> CollectionsHandler.java:237)
> > at
> > org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(
> CollectionsHandler.java:215)
> > at
> > org.apache.solr.handler.RequestHandlerBase.handleRequest(
> RequestHandlerBase.java:173)
> > at
> > org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:748)
> > at
> > org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(
> HttpSolrCall.java:729)
> > at org.apache.solr.servlet.HttpSolrCall.call(
> HttpSolrCall.java:510)
> > at
> > org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:361)
> > at
> > org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:305)
> > at
> > org.eclipse.jetty.servlet.ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1691)
> > at
> > org.eclipse.jetty.servlet.ServletHandler.doHandle(
> ServletHandler.java:582)
> > at
> > org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:143)
> > at
> > org.eclipse.jetty.security.SecurityHandler.handle(
> SecurityHandler.java:548)
> > at
> > org.eclipse.jetty.server.session.SessionHandler.
> doHandle(SessionHandler.java:226)
> > at
> > org.eclipse.jetty.server.handler.ContextHandler.
> doHandle(ContextHandler.java:1180)
> > at
> > org.eclipse.jetty.servlet.ServletHandler.doScope(
> ServletHandler.java:512)
> > at
> > org.eclipse.jetty.server.session.SessionHandler.
> doScope(SessionHandler.java:185)
> > at
> > org.eclipse.jetty.server.handler.ContextHandler.
> doScope(ContextHandler.java:1112)
> > at
> > org.eclipse.jetty.server.handler.ScopedHandler.handle(
> 

Re: Migrating from Solr 6.X to Solr 7.X: "non legacy mode coreNodeName missing"

[Erick Erickson]

You may have to set legacyCloud=true in your cluster properties. In
the Solr reference guide (7.0) there's a section "Changes to Default
Behaviors" that'll tell you a bit about that. Check your cluster
properties (top level znode in ZooKeeper).

Or, try the MIGRATESTATEFORMAT collection API on your 6x collection
first. If your current installation has a single clusterstate.json
that contains all of your collection information, then this would be
indicated.

Mind you I'd experiment on a non-prod system first

Best,
Erick



On Mon, Oct 30, 2017 at 8:44 AM, Dave Seltzer  wrote:
> I have a Solr collection with 1 shard and around 40 replicas hosted on Solr
> 6.6.0
>
> I added a Solr 7.1 server to the cluster and then tried to add a replica
> (ADDREPLICA) onto the new server but got this message:
>
>> Error CREATEing SolrCore 'content_collection_20171013_shard1_replica37':
> non legacy mode coreNodeName missing {collection.configName=content,
> shard=shard1, collection=content_collection_20171013}
>
> Is there something I have to do to prepare this collection for Solr 7.x?
>
> Thanks,
>
> -Dave
>
> [root@crompcoreph02 ~]# curl "http://
> [solrclusterloadbalancer]/solr/admin/collections?action=ADDREPLICA=content_collection_20171013=shard1=$crompcoreph02:8983_solr"
> 
> 
> 500 name="QTime">222 name="crompcoreph02:8983_solr">org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error
> from server at http://crompcoreph02:8983/solr: Error CREATEing SolrCore
> 'content_collection_20171013_shard1_replica37': non legacy mode
> coreNodeName missing {collection.configName=content, shard=shard1,
> collection=content_collection_20171013}org.apache.solr.common.SolrException:org.apache.solr.common.SolrException:
> ADDREPLICA failed to create replica name="msg">ADDREPLICA failed to create replica name="rspCode">500 name="error-class">org.apache.solr.common.SolrException name="root-error-class">org.apache.solr.common.SolrException name="msg">ADDREPLICA failed to create replica name="trace">org.apache.solr.common.SolrException: ADDREPLICA failed to
> create replica
> at
> org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:300)
> at
> org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:237)
> at
> org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:215)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
> at
> org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:748)
> at
> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:729)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:510)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.eclipse.jetty.server.Server.handle(Server.java:534)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
> at
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
> 

Migrating from Solr 6.X to Solr 7.X: "non legacy mode coreNodeName missing"

[Dave Seltzer]

I have a Solr collection with 1 shard and around 40 replicas hosted on Solr
6.6.0

I added a Solr 7.1 server to the cluster and then tried to add a replica
(ADDREPLICA) onto the new server but got this message:

> Error CREATEing SolrCore 'content_collection_20171013_shard1_replica37':
non legacy mode coreNodeName missing {collection.configName=content,
shard=shard1, collection=content_collection_20171013}

Is there something I have to do to prepare this collection for Solr 7.x?

Thanks,

-Dave

[root@crompcoreph02 ~]# curl "http://
[solrclusterloadbalancer]/solr/admin/collections?action=ADDREPLICA=content_collection_20171013=shard1=$crompcoreph02:8983_solr"


500222org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error
from server at http://crompcoreph02:8983/solr: Error CREATEing SolrCore
'content_collection_20171013_shard1_replica37': non legacy mode
coreNodeName missing {collection.configName=content, shard=shard1,
collection=content_collection_20171013}org.apache.solr.common.SolrException:org.apache.solr.common.SolrException:
ADDREPLICA failed to create replicaADDREPLICA failed to create replica500org.apache.solr.common.SolrExceptionorg.apache.solr.common.SolrExceptionADDREPLICA failed to create replicaorg.apache.solr.common.SolrException: ADDREPLICA failed to
create replica
at
org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:300)
at
org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:237)
at
org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:215)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
at
org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:748)
at
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:729)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:510)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
500

Re: In which order are commitwithin delete/add executed?

[David Svånå]

Perfect! Thanks Erick.

On Mon, Oct 30, 2017 at 3:57 PM, Erick Erickson 
wrote:

> Solr executes the deletes and adds in the order they are received.
> Commitwithin isn't relevant to the execution order.
>
> commitWithin just controls when the accumulated changes are flushed to
> a new index segment and is global,
> not per request.
>
> Let's say Solr gets a request with commitWithin. It starts a timer for
> that core. When that timer expires, all
> accumulated updates (adds, deletes, whatever) are written out. For example
>
> T+0 add received with commitWithin set to 5 seconds, add for doc1
> T+1 delete for doc1 with commitWithin set for 5 seconds
> T+5 both the add and delete are written
>
> Best,
> Erick
>
> On Mon, Oct 30, 2017 at 4:35 AM, David Svånå 
> wrote:
> > Hi,
> >
> > If I send multiple instructions /update with a bunch of deletes and adds,
> > all having commitWithin specified, how can I make sure that the
> > instructions are executed in the same order as I send them in?
> >
> > For example, if both a delete and an add is sent for the same ID, I want
> to
> > be able to control whether the delete happens before or after the add. If
> > the 'add' is more recent then I want to run the delete first (or only
> add),
> > or if the add is older, then I should run the delete.
>

Re: In which order are commitwithin delete/add executed?

[Erick Erickson]

Solr executes the deletes and adds in the order they are received.
Commitwithin isn't relevant to the execution order.

commitWithin just controls when the accumulated changes are flushed to
a new index segment and is global,
not per request.

Let's say Solr gets a request with commitWithin. It starts a timer for
that core. When that timer expires, all
accumulated updates (adds, deletes, whatever) are written out. For example

T+0 add received with commitWithin set to 5 seconds, add for doc1
T+1 delete for doc1 with commitWithin set for 5 seconds
T+5 both the add and delete are written

Best,
Erick

On Mon, Oct 30, 2017 at 4:35 AM, David Svånå  wrote:
> Hi,
>
> If I send multiple instructions /update with a bunch of deletes and adds,
> all having commitWithin specified, how can I make sure that the
> instructions are executed in the same order as I send them in?
>
> For example, if both a delete and an add is sent for the same ID, I want to
> be able to control whether the delete happens before or after the add. If
> the 'add' is more recent then I want to run the delete first (or only add),
> or if the add is older, then I should run the delete.

Re: Graph Traversal

[Pratik Patel]

You use this in query time. Since Streaming Expressions can be pipelined,
the next stage/function of pipeline will work on the new tuples generated.

On Mon, Oct 30, 2017 at 10:09 AM, Kojo  wrote:

> Do you store this new tuples, created by Streaming Expressions, in a new
> Solr cloud collection? Or just use this tuples in query time?
>
> 2017-10-30 11:00 GMT-02:00 Pratik Patel :
>
> > By including Cartesian function in Streaming Expression pipeline, you can
> > convert a tuple having one multivalued field into multiple tuples where
> > each tuple holds one value for the field which was originally
> multivalued.
> >
> > For example, if you have following document.
> >
> > { id: someID, fruits: [apple, organge, banana] }   // fruits is
> multivalued
> > > field
> >
> >
> > Applying Cartesian function would give following tuples.
> >
> > { id: someID , fruits: apple }, { id: someID, fruits: orange }, {id:
> > > someID, fruits: banana }
> >
> >
> > Now that fruits holds single values, you can also use any Streaming
> > Expression functions which don't work with multivalued fields. This
> happens
> > in the Streaming Expression pipeline so you don't have to flatten your
> > documents in index.
> >
> > On Mon, Oct 30, 2017 at 8:39 AM, Kojo  wrote:
> >
> > > Hi,
> > > just a question, I have no deep background on Solr, Graph etc.
> > > This solution looks like normalizing data like a m2m table in sql
> > database,
> > > is it?
> > >
> > >
> > >
> > > 2017-10-29 21:51 GMT-02:00 Pratik Patel :
> > >
> > > > For now, you can probably use Cartesian function of Streaming
> > Expressions
> > > > which Joel implemented to solve the same problem.
> > > >
> > > > https://issues.apache.org/jira/browse/SOLR-10292
> > > > http://joelsolr.blogspot.com/2017/03/streaming-nlp-is-
> > > > coming-in-solr-66.html
> > > >
> > > > Regards,
> > > > Pratik
> > > >
> > > > On Sat, Oct 28, 2017 at 7:38 PM, Joel Bernstein 
> > > > wrote:
> > > >
> > > > > I don't see a jira ticket for this yet. Feel free to create it and
> > > reply
> > > > > back with the link.
> > > > >
> > > > > Joel Bernstein
> > > > > http://joelsolr.blogspot.com/
> > > > >
> > > > > On Fri, Oct 27, 2017 at 9:55 AM, Kojo  wrote:
> > > > >
> > > > > > Hi, I was looking for information on Graph Traversal. More
> > > > specifically,
> > > > > > support to search graph on multivalued field.
> > > > > >
> > > > > > Searching on the Internet, I found a question exactly the same of
> > > mine,
> > > > > > with an answer that what I need is not implemented yet:
> > > > > > http://lucene.472066.n3.nabble.com/Using-multi-valued-
> > > > > > field-in-solr-cloud-Graph-Traversal-Query-td4324379.html
> > > > > >
> > > > > >
> > > > > > Is there a ticket on Jira to follow the implementation of search
> > > graph
> > > > on
> > > > > > multivalued field?
> > > > > >
> > > > > > Thank you,
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Graph Traversal

[Kojo]

Do you store this new tuples, created by Streaming Expressions, in a new
Solr cloud collection? Or just use this tuples in query time?

2017-10-30 11:00 GMT-02:00 Pratik Patel :

> By including Cartesian function in Streaming Expression pipeline, you can
> convert a tuple having one multivalued field into multiple tuples where
> each tuple holds one value for the field which was originally multivalued.
>
> For example, if you have following document.
>
> { id: someID, fruits: [apple, organge, banana] }   // fruits is multivalued
> > field
>
>
> Applying Cartesian function would give following tuples.
>
> { id: someID , fruits: apple }, { id: someID, fruits: orange }, {id:
> > someID, fruits: banana }
>
>
> Now that fruits holds single values, you can also use any Streaming
> Expression functions which don't work with multivalued fields. This happens
> in the Streaming Expression pipeline so you don't have to flatten your
> documents in index.
>
> On Mon, Oct 30, 2017 at 8:39 AM, Kojo  wrote:
>
> > Hi,
> > just a question, I have no deep background on Solr, Graph etc.
> > This solution looks like normalizing data like a m2m table in sql
> database,
> > is it?
> >
> >
> >
> > 2017-10-29 21:51 GMT-02:00 Pratik Patel :
> >
> > > For now, you can probably use Cartesian function of Streaming
> Expressions
> > > which Joel implemented to solve the same problem.
> > >
> > > https://issues.apache.org/jira/browse/SOLR-10292
> > > http://joelsolr.blogspot.com/2017/03/streaming-nlp-is-
> > > coming-in-solr-66.html
> > >
> > > Regards,
> > > Pratik
> > >
> > > On Sat, Oct 28, 2017 at 7:38 PM, Joel Bernstein 
> > > wrote:
> > >
> > > > I don't see a jira ticket for this yet. Feel free to create it and
> > reply
> > > > back with the link.
> > > >
> > > > Joel Bernstein
> > > > http://joelsolr.blogspot.com/
> > > >
> > > > On Fri, Oct 27, 2017 at 9:55 AM, Kojo  wrote:
> > > >
> > > > > Hi, I was looking for information on Graph Traversal. More
> > > specifically,
> > > > > support to search graph on multivalued field.
> > > > >
> > > > > Searching on the Internet, I found a question exactly the same of
> > mine,
> > > > > with an answer that what I need is not implemented yet:
> > > > > http://lucene.472066.n3.nabble.com/Using-multi-valued-
> > > > > field-in-solr-cloud-Graph-Traversal-Query-td4324379.html
> > > > >
> > > > >
> > > > > Is there a ticket on Jira to follow the implementation of search
> > graph
> > > on
> > > > > multivalued field?
> > > > >
> > > > > Thank you,
> > > > >
> > > >
> > >
> >
>

Re: Graph Traversal

[Pratik Patel]

By including Cartesian function in Streaming Expression pipeline, you can
convert a tuple having one multivalued field into multiple tuples where
each tuple holds one value for the field which was originally multivalued.

For example, if you have following document.

{ id: someID, fruits: [apple, organge, banana] }   // fruits is multivalued
> field


Applying Cartesian function would give following tuples.

{ id: someID , fruits: apple }, { id: someID, fruits: orange }, {id:
> someID, fruits: banana }


Now that fruits holds single values, you can also use any Streaming
Expression functions which don't work with multivalued fields. This happens
in the Streaming Expression pipeline so you don't have to flatten your
documents in index.

On Mon, Oct 30, 2017 at 8:39 AM, Kojo  wrote:

> Hi,
> just a question, I have no deep background on Solr, Graph etc.
> This solution looks like normalizing data like a m2m table in sql database,
> is it?
>
>
>
> 2017-10-29 21:51 GMT-02:00 Pratik Patel :
>
> > For now, you can probably use Cartesian function of Streaming Expressions
> > which Joel implemented to solve the same problem.
> >
> > https://issues.apache.org/jira/browse/SOLR-10292
> > http://joelsolr.blogspot.com/2017/03/streaming-nlp-is-
> > coming-in-solr-66.html
> >
> > Regards,
> > Pratik
> >
> > On Sat, Oct 28, 2017 at 7:38 PM, Joel Bernstein 
> > wrote:
> >
> > > I don't see a jira ticket for this yet. Feel free to create it and
> reply
> > > back with the link.
> > >
> > > Joel Bernstein
> > > http://joelsolr.blogspot.com/
> > >
> > > On Fri, Oct 27, 2017 at 9:55 AM, Kojo  wrote:
> > >
> > > > Hi, I was looking for information on Graph Traversal. More
> > specifically,
> > > > support to search graph on multivalued field.
> > > >
> > > > Searching on the Internet, I found a question exactly the same of
> mine,
> > > > with an answer that what I need is not implemented yet:
> > > > http://lucene.472066.n3.nabble.com/Using-multi-valued-
> > > > field-in-solr-cloud-Graph-Traversal-Query-td4324379.html
> > > >
> > > >
> > > > Is there a ticket on Jira to follow the implementation of search
> graph
> > on
> > > > multivalued field?
> > > >
> > > > Thank you,
> > > >
> > >
> >
>

Re: Graph Traversal

[Kojo]

Hi,
just a question, I have no deep background on Solr, Graph etc.
This solution looks like normalizing data like a m2m table in sql database,
is it?



2017-10-29 21:51 GMT-02:00 Pratik Patel :

> For now, you can probably use Cartesian function of Streaming Expressions
> which Joel implemented to solve the same problem.
>
> https://issues.apache.org/jira/browse/SOLR-10292
> http://joelsolr.blogspot.com/2017/03/streaming-nlp-is-
> coming-in-solr-66.html
>
> Regards,
> Pratik
>
> On Sat, Oct 28, 2017 at 7:38 PM, Joel Bernstein 
> wrote:
>
> > I don't see a jira ticket for this yet. Feel free to create it and reply
> > back with the link.
> >
> > Joel Bernstein
> > http://joelsolr.blogspot.com/
> >
> > On Fri, Oct 27, 2017 at 9:55 AM, Kojo  wrote:
> >
> > > Hi, I was looking for information on Graph Traversal. More
> specifically,
> > > support to search graph on multivalued field.
> > >
> > > Searching on the Internet, I found a question exactly the same of mine,
> > > with an answer that what I need is not implemented yet:
> > > http://lucene.472066.n3.nabble.com/Using-multi-valued-
> > > field-in-solr-cloud-Graph-Traversal-Query-td4324379.html
> > >
> > >
> > > Is there a ticket on Jira to follow the implementation of search graph
> on
> > > multivalued field?
> > >
> > > Thank you,
> > >
> >
>

In which order are commitwithin delete/add executed?

[David Svånå]

Hi,

If I send multiple instructions /update with a bunch of deletes and adds,
all having commitWithin specified, how can I make sure that the
instructions are executed in the same order as I send them in?

For example, if both a delete and an add is sent for the same ID, I want to
be able to control whether the delete happens before or after the add. If
the 'add' is more recent then I want to run the delete first (or only add),
or if the add is older, then I should run the delete.

Re: SolrCloud not able to view cloud page - Loading of "/solr/zookeeper?wt=json" failed (HTTP-Status 500)

[Ere Maijala]

On the Solr side there's at least 
https://issues.apache.org/jira/browse/SOLR-9818 which may cause trouble 
with the queue. I once had the core reload command in the admin UI add 
more than 200k entries to the overseer queue..


--Ere

Shawn Heisey kirjoitti 25.10.2017 klo 15.57:

On 10/24/2017 8:11 AM, Tarjono, C. A. wrote:

Would like to check if anyone have seen this issue before, we started
having this a few days ago:

  


The only error I can see in solr console is below:

5960847[main-SendThread(172.16.130.132:2281)] WARN
org.apache.zookeeper.ClientCnxn [ ] – Session 0x65f4e28b7370001 for
server 172.16.130.132/172.16.130.132:2281, unexpected error, closing
socket connection and attempting reconnect java.io.IOException: Packet
len30829010 is out of range!



Combining the last part of what I quoted above with the image you shared
later, I am pretty sure I know what is happening.

The overseer queue in zookeeper (at the ZK path of /overseer/queue) has
a lot of entries in it.  Based on the fact that you are seeing a packet
length beyond 30 million bytes, I am betting that the number of entries
in the queue is between 1.5 million and 2 million.  ZK cannot handle
that packet size without a special startup argument.  The value of the
special parameter defaults to a little over one million bytes.

To fix this, you're going to need to wipe out the overseer queue.  ZK
includes a script named ZkCli.  Note that Solr includes a script called
zkcli as well, which does very different things.  You need the one
included with zookeeper.

Wiping out the queue when it is that large is not straightforward.  You
need to start the ZkCli script included with zookeeper with a
-Djute.maxbuffer=3100 argument and the same zkHost value used by
Solr, and then use a command like "rmr /overseer/queue" in that command
shell to completely remove the /overseer/queue path.  Then you can
restart the ZK servers without the jute.maxbuffer setting.  You may need
to restart Solr.  Running this procedure might also require temporarily
restarting the ZK servers with the same jute.maxbuffer argument, but I
am not sure whether that is required.

The basic underlying problem here is that ZK allows adding new nodes
even when the size of the parent node exceeds the default buffer size.
That issue is documented here:

https://issues.apache.org/jira/browse/ZOOKEEPER-1162

I can't be sure why why your cloud is adding so many entries to the
overseer queue.  I have seen this problem happen when restarting a
server in the cloud, particularly when there are a large number of
collections or shard replicas in the cloud.  Restarting multiple servers
or restarting the same server multiple times without waiting for the
overseer queue to empty could also cause the issue.

Thanks,
Shawn



--
Ere Maijala
Kansalliskirjasto / The National Library of Finland