Re: Congratulations to the new Apache Solr PMC Chair, Jan Høydahl!

Congrats Jan and Best Wishes! Also many thanks to Anshum for all his efforts in the last term! Also a belated shout out to Cassandra and other chairs in the past for their much appreciated efforts for the community ! Regards Aroop > On Feb 18, 2021, at 10:55 AM, Anshum Gupta wrote: > > Hi

Re: UPDATE collection's Rule-based Replica Placement

Moshe An indirect way to do this could be to take backup of this collection and then restore with the desired placement rules. Backup: Example: curl “https://solr.foo.com/solr/admin/collections?

Re: BasicAuth help

Try looking at a simple ldap authentication suggested here: https://github.com/itzmestar/ldap_solr You can combine this for authentication and couple it with rule based authorization. > On Aug 28, 2020, at 12:26 PM, Vanalli, Ali A - DOT

Re: Solr with HDFS configuration example running in production/dev

HDFS will still be there, just NOT on the core package, but as a plug-in or contrib. > On Aug 20, 2020, at 11:07 AM, Aroop Ganguly wrote: > > HDFS will still be there, just on the core package, but as a plug-in or > contrib.

Re: Solr with HDFS configuration example running in production/dev

HDFS will still be there, just on the core package, but as a plug-in or contrib. > On Aug 19, 2020, at 7:50 AM, Prashant Jyoti wrote: > > You're right Andrew. Even I read about that. But there's a use case for > which we want to configure the said case. > > Are you also aware of what feature

Re: SOLR indexing takes longer time

Adding on to what others have said, indexing speed in general is largely affected by the parallelism and isolation you can give to each node. Is there a reason why you cannot have more than 1 shard? If you have 5 node cluster, why not have 5 shards, maxshardspernode=1 replica=1 is ok. You should

Re: Multiple Collections in a Alias.

ion sync script needs some debugging most likely :) > On Aug 12, 2020, at 4:29 PM, Jae Joo wrote: > > Good question. How can I validate if the replicas are all synched? > > > On Wed, Aug 12, 2020 at 7:28 PM Jae Joo wrote: > >> numFound is same but different score. >>

Re: Multiple Collections in a Alias.

: > >> Good question. How can I validate if the replicas are all synched? >> >> >> On Wed, Aug 12, 2020 at 7:28 PM Jae Joo wrote: >> >>> numFound is same but different score. >>> >>> >>> >>> >>>

Re: Multiple Collections in a Alias.

Joo wrote: > > The replications are all synched and there are no updates while I was > testing. > > > On Wed, Aug 12, 2020 at 1:49 PM Aroop Ganguly > wrote: > >> Most likely you have 1 or more collections behind the alias that have >> replicas out of sync

Re: Multiple Collections in a Alias.

Most likely you have 1 or more collections behind the alias that have replicas out of sync :) Try querying each collection to find the one out of sync. > On Aug 12, 2020, at 10:47 AM, Jae Joo wrote: > > I have 10 collections in single alias and having different result sets for > every time

Re: Cannot add replica during backup

the data in > 15 min! > >>> It would also help to ensure your overseer is on a node with a role that > exempts it from any Solr index responsibilities. > How would I ensure this? First I'm hearing about this! > > Thanks for all the help!! > > On Tue, Aug 11, 20

Re: Solr + Parquets

> script to iterate and load the files via the post command. You mean load parquet filed over post? That sounds unbelievable … Do u mean you created Solr doc for each parquet record in a partition and used solrJ or some other java lib to post the docs to Solr? df.mapPatitions(p => { ///batch

Re: Cannot add replica during backup

ably have to see which part of the process is the slowest. > > On another note, can you simply remove the task from the ZK path to > continue the execution of tasks? > > Regards, > > Ash > > On Tue, Aug 11, 2020 at 11:40 AM Aroop Ganguly > wrote: > >> 12 hou

Re: Cannot add replica during backup

12 hours is extreme, we take backups of 10TB worth of indexes in 15 mins using the collection backup api. How are you taking the backup? Do you actually see any backup progress or u are just seeing the task in the overseer queue linger ? I have seen restore tasks hanging in the queue forever

Re: Solr Backup/Restore

Restore will only create the same number of shards as the original collection had when you took the backup. If you are on a cluster with enough resources, you can try split shards to the desired numbers later on? Split Shards has a more efficient implementation in solr 8.x but if u have a

Re: [ANNOUNCE] Apache Solr 8.6.0 released

t; >> Perhaps the deprecation notices should feature on >> https://lucene.apache.org/solr/news.html ? Because right now, they're not >> *very >> *visible in the changes. >> >> On Thu, 16 Jul 2020 at 01:18, Aroop Ganguly > .invalid> >> wrote: >

Re: [ANNOUNCE] Apache Solr 8.6.0 released

May we ask what in hdfs support is being deprecated? Is Hdfs backup and restore being deprecated ? Sent from my iPhone > On Jul 15, 2020, at 3:41 PM, Houston Putman wrote: > > To address your concern Bernd, > > The goal of this deprecation is not to remove the functionality entirely. > The

Re: [EXTERNAL] - Re: HTTP 401 when searching on alias in secured Solr

Isabele sometime 401’s are a red herring for other issues un related to auth. We have had issues on 7.7 where an underlying transient replica recovery and/or leader down situation where the only message we got back from Solr was a 401. Please see if u have any down replicas or other issues where

Re: Solr Streaming Expression failures

I have personally not used streaming expressions to commit data to a collection (have used them a lot of querying), and would not recommend it for bulk indexing unless Joel recommends it :) On the other hand we have had decent success in indexing at scale and 12 million is not a big number.

Re: How do *you* restrict access to Solr?

Hi Ryan You should consider a simple rule based authorization scheme. Your staff user can be given readonly privileges to everything you want to except the admin ui. Depending on which version of solr you are on this can be trivial. - Aroop > On Mar 16, 2020, at 8:46 AM, Ryan W wrote: > >

Re: Overseer & Backups - Questions

xpected when the backup completed a few hours > ago, the task then got completed. Is there some concurrency setting with > these tasks? Or is a backup a blocking task? We noticed that the index was > still being flushed to segments though. > > Regards, > > Ash > > On Wed

Re: Overseer & Backups - Questions

May we know how you are invoking backups ? > On Mar 9, 2020, at 11:53 PM, Ashwin Ramesh wrote: > > Hi everybody, > > Quick Specs: > - Solr 7.4 Solr Cloud > - 30gb index on 8 shards Tlog/Pull > > We run daily backups on our 30gb index and noticed that the overseer does > not process other jobs

Re: Backups with only 1 machine having access to remote storage?

Hi Koen Which backup mechanism are you using ? HDFS backup setup is a lot more sophisticated, and backup repository settings made in the solr.xml manage lots of these things. The node from where you issue the command would not have any bearing on the target collections’s data that you are

Restore from HDFS slow

Hey Solr Experts Anyone has idea how to ensure restore collections from HDFS can be made faster? Are there any tuning parameters like number of threads to use, the memory to use etc, that can be configured someplace to enhance/manage the restore process? I am on Solr 7.7.2 btw and the api we

Re: alias read access impossible for anyone other than admin?

a JIRA for. RBAP > permission matching/processing can be subtle for those using it for > the first time, so any improvement to the docs will go a long way. > > Jason > > On Sat, May 25, 2019 at 3:12 AM Aroop Ganguly wrote: >> >> hi jason >> >> which vers

Re: alias read access impossible for anyone other than admin?

hi jason which version of solr has the definitive fix for the rbap again ? also is there a jira to fix or create a documentation for the same that works :) ? aroop > On May 24, 2019, at 9:55 AM, Jason Gerlowski wrote: > > Hi Sotiris, > > First, what version of Solr are you running? We've

collection exists but delete by query fails

Hi I am on Solr 7.5 and I am issuing a delete-by-query using CloudSolrClient The collection exists but issuing a deletebyquery is failing every single time. I am wondering what is happening, and how to debug this. org.apache.solr.client.solrj.SolrServerException:

Re: Solr 7.5 - Indexing Failing due to "IndexWriter is Closed"

org/ (my blog) > >> On Apr 2, 2019, at 8:13 PM, Aroop Ganguly wrote: >> >> Mutliple threads to the same index ? And how many concurrent threads? >> >> Our case is not merely multiple threads but actually large scale spark >> indexer jobs that index 1B record

Re: Slower indexing speed in Solr 8.0.0

Indexing speeds are function of a lot of variables in my experience. What is your setup like? What kind of cluster you have, the number of shards you created, the number of machines etc? Where is your input data coming from? What technology do you use to indexing (simple java threads or

Re: Solr 7.5 - Indexing Failing due to "IndexWriter is Closed"

bserver.wunderwood.org/ (my blog) > >> On Apr 1, 2019, at 10:51 PM, Aroop Ganguly wrote: >> >> Turns out the cause was multiple indexing jobs indexing into the index >> simultaneously, which one can imagine can cause jvm loads on certain >> replicas for sure.

Re: IndexWriter has closed

; Regards, > Edwin > > On Fri, 29 Mar 2019 at 15:16, Aroop Ganguly wrote: > >> Trying again .. Any idea why this might happen? >> >> >>> On Mar 27, 2019, at 10:43 PM, Aroop Ganguly >> wrote: >>> >>> Hi Everyone >>> >&

Re: Solr 7.5 - Indexing Failing due to "IndexWriter is Closed"

trace! > On Apr 1, 2019, at 5:32 PM, Shawn Heisey wrote: > > 4/1/2019 5:40 PM, Aroop Ganguly wrote: >> Thanks Shawn, for the initial response. >> Digging into a bit, I was wondering if we’d care to read the inner most >> stack. >> From the inner most stack it

Re: Solr 7.5 - Indexing Failing due to "IndexWriter is Closed"

) at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68) > On Apr 1, 2019, at 4:06 PM, Shawn Heisey wrote: > > On 4/1/2019 4:44 PM, Aroop Ganguly wrote: >> I am facing this issue again.The stack mentions Heap space issue. >

Solr 7.5 - Indexing Failing due to "IndexWriter is Closed"

Hi Group I am facing this issue again.The stack mentions Heap space issue. Are the document sizes too big ? Not sure what I should be doing here; As on the solr admin ui I do not see jvm being anywhere close to being full. Any advise on this is greatly welcome. Full Stack trace:

Solr 7.5 - Indexing Failing due to "IndexWriter is Closed"

Hi Group I am facing this issue again.The stack mentions Heap space issue. Are the document sizes too big ? Not sure what I should be doing here; As on the solr admin ui I do not see jvm being anywhere close to being full. Any advise on this is greatly welcome. Full Stack trace: 2019-04-01

Re: IndexWriter has closed

Trying again .. Any idea why this might happen? > On Mar 27, 2019, at 10:43 PM, Aroop Ganguly wrote: > > Hi Everyone > > My indexing jobs are failing with “this IndexWriter has closed” errors.. > This is a solr 7.5 setup, with an NRT index. > > In deeper logs I see,

IndexWriter has closed

Hi Everyone My indexing jobs are failing with “this IndexWriter has closed” errors.. This is a solr 7.5 setup, with an NRT index. In deeper logs I see, some of these exceptions, Any idea what could have caused this ? o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException:

Re: Solr 7.5 DeleteShard not working when all cores are down

shards with all replicas inactive in state. > On Mar 14, 2019, at 8:01 PM, Aroop Ganguly wrote: > > Thanks Shalin, Shawn. > > I ended up getting guidance from Anshum on this and we did indeed use the > delete-replica api to delete all but one of the replicas, and bouncing t

Solr 7.5 DeleteShard not working when all cores are down

Hi All I am trying to delete a shard from a collection using the collections api for the same. On the solr ui, all the replicas are in “downed” state. However, when I run the delete shard command: /solr/admin/collections?action=DELETESHARD=x=shard84 I get this exception: {

Re: solr 7 optimize with Tlog/Pull replicas

ugh faster to matter in your particular situation, then the cost may be > worth it. And the situation where it makes the most sense is situations where > you can optimize regularly. > > Best, > Erick > >> On Mar 12, 2019, at 10:51 PM, Aroop Ganguly >>

Re: solr 7 optimize with Tlog/Pull replicas

Hi Erick A related question: Is optimize then ill advised for bulk indexer post solr 7.5 ? >> Especially in a situation where an index is being modified over many days ? Thanks Aroop > On Mar 12, 2019, at 9:30 PM, Wei wrote: > > Thanks Erick, it's very helpful. So for bulking indexing in

Re: Hide BasicAuth JVM param on SOLR admin UI

try changing the passwords using the auth api https://lucene.apache.org/solr/guide/6_6/basic-authentication-plugin.html#BasicAuthenticationPlugin-AddaUserorEditaPassword

Re: Rule Based Auth - Permission to run select queries

I guess just mentioning path=“/select” is sufficient… The documentation does not explain this .. > On Mar 5, 2019, at 7:36 PM, Aroop Ganguly wrote: > > Turns out I had to specify the path param to /select, while setting the > permission. > > But this is random..I create

Re: Rule Based Auth - Permission to run select queries

uot;, "role":"readonly","path":"/select"} How does this work ? Is there actually a permission called read-collections? > On Mar 5, 2019, at 7:08 PM, Aroop Ganguly wrote: > > Hi Team > > I am playing around with rule based auth and I wan

Re: RuleBasedAuthorizationPlugin configuration

Hi Dominique Were you able to resolve this ? I am also stuck with understanding a minimal permission-set to give to a readonly user to read from the /select endpoint. Regards Aroop > On Jan 1, 2019, at 11:23 PM, Dominique Bejean > wrote: > > Hi, > > I created a Jira issue >

Rule Based Auth - Permission to run select queries

Hi Team I am playing around with rule based auth and I wanted to create a role which is readonly. I gave the “read” permission to the role, but I am not able to get data from the /select handler. A simple select request results in this response: Error 403 Unauthorized request, Response

Re: Unable to create collection with custom queryParser Plugin

ed to put them in the fat jar. >> >>> Am 11.02.2019 um 05:59 schrieb Aroop Ganguly : >>> >>> Thanks Erick! >>> >>> I see. Yes it is a fat jar post shadowJar process (in the order of MBs). >>> It contains solrj and solr-core dependencie

Re: Unable to create collection with custom queryParser Plugin

an, say, > 15K it's a red flag. If you do a "jar -dvf your_custom_jar" there > should be _very_ few classes in it. > > Best, > Erick > > On Sun, Feb 10, 2019 at 8:33 PM Aroop Ganguly > wrote: >> >> [resending due to bounce warning from the other em

Unable to create collection with custom queryParser Plugin

) at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:792) ... 51 more  Aroop Ganguly Siri Data | Metrics Platform

Unable to create collection with custom queryParser Plugin

Hi Team I thought this was simple, but I am just missing something here. Any guidance would be very appreciated. What have I done so far: 1. I have created a custom querParser (class SamplePluggin extends QParserPlugin { ), which right now does nothing but logs an info message, and

Re: streaming expressions substring-evaluator

lations. >>> >>> Joel Bernstein >>> http://joelsolr.blogspot.com/ >>> >>> >>> On Wed, Oct 31, 2018 at 2:37 AM Aroop Ganguly >>> wrote: >>> >>>> Hey Team >>>> >>>> >>>> Is the

streaming expressions substring-evaluator

Hey Team Is there a way to extract a part of a string field and group by on it and obtain a histogram ? for example the filed value is DateTime of the form: 20180911T00 and I want to do a substring like substring(field1,0,7), and then do a streaming expression of the form : rollup(

Re: Silk from LucidWorks

How do you use Grafana with Solr ? Did you build a http communication interface or is there some open source project that you leveraged ? > On Jul 15, 2018, at 2:54 PM, Rahul Singh wrote: > > Their commercial offering still has something like it. You can always try > Grafana > > Rahul > On

Re: Text Similarity

ld be joined again. > > Rahul > On Jul 11, 2018, 4:42 PM -0400, Aroop Ganguly > , wrote: >> Hi Team >> >> This is what I want to do: >> 1. I have 2 datasets of the schema id-number and company-name >> 2. I want to ultimately be able to link (join or any

Text Similarity

Hi Team This is what I want to do: 1. I have 2 datasets of the schema id-number and company-name 2. I want to ultimately be able to link (join or any other means) the 2 data sets based on the similarity between the company-name fields of the 2 data set. Example: Dataset 1 Id | Company

Re: String concatenation in Streaming Expressions

type:"Article",fq=tags:"genetics", > qt="/export"), > conceptid as conceptid, > storeid as "test_", > concat([conceptid,storeid], conceptid, "-") > ) > > It generates an exception, "Invalid expression > concat([conceptid,st

Re: String concatenation in Streaming Expressions

6_4/solr/solrj/src/test/org/apache/solr/client/solrj/io/stream/ops/ConcatOperationTest.java> > On Jun 27, 2018, at 1:27 PM, Aroop Ganguly wrote: > > It should, but 6.6.* has some issues of things not working per documentation. > Try using 7+. > >> On Jun 27, 2018, at 1

Re: String concatenation in Streaming Expressions

; On Wed, Jun 27, 2018 at 4:18 PM, Aroop Ganguly > wrote: > >> So it will become: >> select( >> search(..), >> conceptid as foo, >> storeid as bar >> append(conceptid, storeid) as id >> ) >> >> Or >> select >> selec

Re: String concatenation in Streaming Expressions

So it will become: select( search(..), conceptid as foo, storeid as bar append(conceptid, storeid) as id ) Or select select( search(..), conceptid as foo, storeid as bar ), foo, bar, append(foo,bar) as id ) > On Jun 27, 2018, at 1:12 PM, Aroop Ganguly wrote: > >

Re: String concatenation in Streaming Expressions

olr/client/solrj/io/stream/eval/AppendEvaluatorTest.java> > On Jun 27, 2018, at 1:07 PM, Aroop Ganguly wrote: > > I think u can use the append evaluator > https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/AppendEvaluator

Re: String concatenation in Streaming Expressions

I think u can use the append evaluator https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/io/eval/AppendEvaluator.java

Re: Java library for building Streaming Expressions

From my experience this is still limited and lot of things are broken if you still want to do it in a strongly typed manner. The only reliable way is to go right now is via a string-expression building route for the streaming expression. That being said, if your expressions are simple enough

Re: Total Collection Size in Solr 7

Ah ok ! > On Jun 27, 2018, at 8:53 AM, Erick Erickson wrote: > > Just sum up the sizes of all the files in your index directory. Clumsy > to be sure > > On Tue, Jun 26, 2018 at 3:12 PM, Aroop Ganguly > wrote: >> Hi Eric >> >> Thanks for the ad

Re: Total Collection Size in Solr 7

0% stored data? Those ratios have huge implications > on whether you're straining anything except disk space. > > There are a lot of metrics, starting with Solr 6.4 that are available > that give you a much better view of Solr's health. > > Best, > Erick > > On Tu

Re: Total Collection Size in Solr 7

admin UI, there are several JIRAs. > Perhaps you'd like to join that conversation? We need to have input, > especially in terms of what kinds of information would be useful from > a practitioner's standpoint. > > Best, > Erick > >> On Mon, Jun 25, 2018 at 11:26 PM, Aroop Ganguly >

Re: Total Collection Size in Solr 7

t; i created a utility using solrzkclient api to read state.json, enumerated > (one) replica for each shard and used /replication handler for size and added > them up.. > > Sent from my iPhone > >> On Jun 25, 2018, at 7:24 PM, Aroop Ganguly wrote: >> >>

Re: Indexing Approach

Would you mind sharing details on 1. the Solr Cloud setup, how may nodes do you have at your disposal and how many shards do you have setup ? 2. The indexing technology, what are you using? Core java/.net threads ? Or a system like spark ? 3. Where do you see the exceptions? The indexer process

Total Collection Size in Solr 7

Hi Team I am not sure how to ascertain the total size of a collection via the Solr UI on a Solr7+ installation. The collection is shared and replicated heavily so its tedious to have to look at each core and figure out the size of the entire collection from this in an additive way. Is there

Re: Search streaming expressions returns rows times number of shards docs

So I think 2 things are being missed here. You should be specifying the qt=“/export” to see all the results. If you do not do that, then the select handler is used by default which gives the default 10-20 rows as result. > On Jun 21, 2018, at 12:53 PM, Joel Bernstein wrote: > > That is

Re: Import data from standalone solr into a solrcloud collection

So I will have > to set replicationfactor of 2 & numShards =2 ? > > On Tue, Jun 19, 2018 at 12:46 PM Aroop Ganguly > wrote: > >> Hi Sushant >> >> replicationFactor defaults to 1 and is not mandatory. >> numShards is mandatory, where you’d equate

Re: Import data from standalone solr into a solrcloud collection

Hi Sushant replicationFactor defaults to 1 and is not mandatory. numShards is mandatory, where you’d equate it to 1. Aroop > On Jun 19, 2018, at 12:29 PM, Sushant Vengurlekar > wrote: > > Thank you Eric. > > In the create collection command I need to set the replication factor > though

Re: Solr Odbc for Parallel Sql integration with Tableau

arious Streaming Expression joins. > > Joel Bernstein > http://joelsolr.blogspot.com/ > > On Mon, Jun 18, 2018 at 6:37 PM, Aroop Ganguly > wrote: > >> Ok I was able to setup the odic bridge (using OpenLink) and I see the >> collections popping up in Tableau too. &g

Re: Solr Odbc for Parallel Sql integration with Tableau

in Solr Parallel Sql ? > On Jun 18, 2018, at 12:30 PM, Aroop Ganguly wrote: > > > Hi Everyone > > I am not sure if something has been done on this yet, though I did see a JIRA > with links to the parallel sql documentation, but I do not think that answers > the questi

Solr Odbc for Parallel Sql integration with Tableau

Hi Everyone I am not sure if something has been done on this yet, though I did see a JIRA with links to the parallel sql documentation, but I do not think that answers the question. I love the jdbc driver and it works well for many UIs but there are other systems that need an ODBC driver.