Re: Embedded SOLR - Best practice?
When you say " caching 100.000 docs" what do you mean ? being able to quickly find information in a corpus which increases in size ( 100.000 docs) everyday ? I second Erick, I think this is fairly normal Solr use case. If you really care about fast searches, you will get a fairly acceptable default configuration. Then, you can tune Solr caching if you need. Just remember that nowadays by default Solr is optimized for Near Real Time Search and it vastly uses the Memory Mapping feature of modern OSs. This means that Solr is not going to do I/O all the time with the disk but index portions will be memory mapped (if the memory assigned to the OS is enough on the machine) . Furthemore you may use the heap memory assigned to the Solr JVM to cache additional elements [1] . In conclusion : I never used the embedded Solr Server ( apart from integration tests). If you really want to play a bit with a scenario where you don't need persistency on disk, you may play with the RamDirectory[2], but also in this case, I generally discourage this approach unless very specific usecases and small indexes. [1] https://lucene.apache.org/solr/guide/6_6/query-settings-in-solrconfig.html#QuerySettingsinSolrConfig-Caches [2] https://lucene.apache.org/solr/guide/6_6/datadir-and-directoryfactory-in-solrconfig.html#DataDirandDirectoryFactoryinSolrConfig-SpecifyingtheDirectoryFactoryForYourIndex - --- Alessandro Benedetti Search Consultant, R Software Engineer, Director Sease Ltd. - www.sease.io -- Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Embedded SOLR - Best practice?
I don't really understand what you're saying here. Solr is pretty fast, why not just put all 400K docs on a Solr instance and just use that? EmbeddedSolrServer works, but it really seems like for a small corpus like this just using a separate stand-alone Solr is way easier. Best, Erick On Wed, Nov 22, 2017 at 11:39 AM, hvengurlekar <harsh2...@gmail.com> wrote: > Hello Folks, > Currently, I am using SOLR in production and the around 40 documents are > stored. For a particular use-case, I am looking to cache around 10 > documents daily. I am thinking of using embedded solr as the cache to > support the range queries. I did not find any good documentation regarding > why embedded solr is not a best practice? > Could you guys please help me out? Thanks! > > > > -- > Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Embedded SOLR - Best practice?
Hello Folks, Currently, I am using SOLR in production and the around 40 documents are stored. For a particular use-case, I am looking to cache around 10 documents daily. I am thinking of using embedded solr as the cache to support the range queries. I did not find any good documentation regarding why embedded solr is not a best practice? Could you guys please help me out? Thanks! -- Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Unit testing HttpPost With an Embedded Solr Server
Hello,
I have written a data service to send an HttpPost command to post JSON to
Solr. The code is working, but now I want to switch to using an embedded
Solr server for just the unit tests. The problem is that the embedded Solr
server doesn't seem to be starting an embedded server with a port. So I'm
at a loss on how to test this. I guess I have two questions. (1) How do I
unit test my post command with an embedded Solr server? (2) If it isn't
possible to use the embedded Solr server, I believe I read somewhere that
Solr uses a Jetty server. Is it possible to convert an embedded jetty
server (with a port I can access) to a Solr server?
Here is the class I am trying to test:
public class SolrDataServiceClient {
private String urlString;
private HttpClient httpClient;
private final Logger LOGGER = LoggerFactory.getLogger
(SolrDataServiceClient.class);
/**
* Constructor for connecting to the Solr Server
* @param solrCore
* @param serverName
* @param portNumber
*/
public SolrDataServiceClient(String solrCore, String serverName,
String portNumber){
LOGGER.info("Initializing new Http Client to Connect To Solr");
urlString = serverName + ":" + portNumber + "/solr/" + solrCore
;
if(httpClient == null){
httpClient = new HttpClient();
}
}
/**
* Post the provided JSON to Solr
*/
public CloseableHttpResponse postJSON(String jsonToAdd) {
CloseableHttpResponse response = null;
try {
CloseableHttpClient client =
HttpClients.createDefault();
HttpPost httpPost = new HttpPost(urlString +
"/update/json/docs");
HttpEntity entity = new ByteArrayEntity(jsonToAdd
.getBytes("UTF-8"));
httpPost.setEntity(entity);
httpPost.setHeader("Content-type", "application/json");
LOGGER.debug("httpPost = " + httpPost.toString());
response = client.execute(httpPost);
String result = EntityUtils.toString(response.getEntity
());
LOGGER.debug("result = " + result);
client.close();
} catch (IOException e) {
LOGGER.error("IOException", e);
}
return response;
}
Here is my JUnit test:
public class SolrDataServiceClientTest {
private static EmbeddedSolrServer embeddedServer;
private static SolrDataServiceClient solrDataServiceClient;
@BeforeClass
public static void setUpBeforeClass() throws Exception {
System.setProperty("solr.solr.home", "solr/conf");
System.setProperty("solr.data.dir", new File(
"target/solr-embedded-data").getAbsolutePath());
CoreContainer coreContainer = new CoreContainer("solr/conf");
coreContainer.load();
CoreDescriptor cd = new CoreDescriptor(coreContainer, "myCoreName",
new File("solr").getAbsolutePath());
coreContainer.create(cd);
embeddedServer = new EmbeddedSolrServer(coreContainer, "myCoreName");
solrDataServiceClient = new SolrDataServiceClient("myCoreName",
"http://localhost;, "8983"); //I'm not sure what should go here
}
@Test
public void testPostJson() {
String testJson = " { " +
"\"observationId\": \"12345c\"," +
"\"observationType\": \"image\"," +
"\"locationLat\": 38.9215," +
"\"locationLon\": -77.235" +
"}";
CloseableHttpResponse response = solrDataServiceClient.postJSON(
testJson);
assertEquals(response.getStatusLine().getStatusCode(), 200);
}
Thank you!
Jennifer
Re: Embedded Solr now deprecated?
On 08/05/2015 08:34 PM, Ken Krugler wrote: Hi Shawn, We have a different use case than the ones you covered in your response to Robert (below), which I wanted to call out. We currently use the embedded server when building indexes as part of a Hadoop workflow. The results get copied to a production analytics server and swapped in on a daily basis. Writing to multiple embedded servers (one per reduce task) gives us maximum performance, and has proven to be a very reliable method for the daily rebuild of pre-aggregations we need for our analytics use case. Regards, -- Ken Hi, applications that are ultimately deployed by a third party not well versed in the intricacies of deploying a full-blown installation of solr would be another use case. Embedding the server, either through embedded class or by deploying it entirely within web application (as opposed to a stand alone server) might be preferred as it makes deployment much simpler. There are applications where solr could be used as a search engine to supplement their core functionalities but only when one doesnt have to pay the price of overly complicated deployment of the entire package. lukasz PS - I'm also currently looking at using embedded Solr as a state storage engine for Samza. From: Shawn Heisey Sent: August 5, 2015 7:54:07am PDT To: solr-user@lucene.apache.org Subject: Re: Embedded Solr now deprecated? On 8/5/2015 7:09 AM, Robert Krüger wrote: I tried to upgrade my application from solr 4 to 5 and just now realized that embedded use of solr seems to be on the way out. Is that correct or is there a just new API to use for that? Building on Erick's reply: I doubt that the embedded server is going away, and I do not recall seeing *anything* marking the entire class deprecated. The class still receives attention from devs -- this feature was released with 5.1.0: https://issues.apache.org/jira/browse/SOLR-7307 That said, we have discouraged users from deploying it in production for quite some time, even though it continues to exist and receive developer attention. Some of the reasons that I think users should avoid the embedded server: It doesn't support SolrCloud, you cannot make it fault-tolerant (redundant), and troubleshooting is harder because you cannot connect to it from outside of the source code where it is embedded. Deploying Solr as a network service offers much more capability than you can get when you embed it in your application. Chances are that you can easily replace EmbeddedSolrServer with one of the SolrClient classes and use a separate Solr deployment from your application. Thanks, Shawn -- Ken Krugler +1 530-210-6378 http://www.scaleunlimited.com custom big data solutions training Hadoop, Cascading, Cassandra Solr -- - Lukasz Salwinski PHONE:310-825-1402 UCLA-DOE Institute FAX:310-206-3914 UCLA, Los AngelesEMAIL: luk...@mbi.ucla.edu -
Re: Embedded Solr stopped to index after a while
(shooting in the dark) What does your data directory looks like? File sizes, etc. And which Operating System. 4Gb is when Windows FAT filesystem has a size limit, but it really should not be that. Regards, Alex. Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter: http://www.solr-start.com/ On 6 August 2015 at 11:57, Aldric THOMAZO a.thom...@e-deal.com wrote: Hello, I have an issue with embedded solr or a misconfiguration but no clue to resolve it. Solr stopped indexing a large set of data from a database after a while. It was running for many hours and when it reached a size of 4GB it stopped running although we are expecting about 40GB, without any error. Some data cannot be found from the solr index. Has anyone already faced such a problem? Is it possible to have 40GB or more regarding the data set to index? Thanks in advanced. Regards, Aldric
Embedded Solr stopped to index after a while
Hello, I have an issue with embedded solr or a misconfiguration but no clue to resolve it. Solr stopped indexing a large set of data from a database after a while. It was running for many hours and when it reached a size of 4GB it stopped running although we are expecting about 40GB, without any error. Some data cannot be found from the solr index. Has anyone already faced such a problem? Is it possible to have 40GB or more regarding the data set to index? Thanks in advanced. Regards, Aldric
Embedded Solr now deprecated?
Hi, I tried to upgrade my application from solr 4 to 5 and just now realized that embedded use of solr seems to be on the way out. Is that correct or is there a just new API to use for that? Thanks in advance, Robert
Re: Embedded Solr now deprecated?
Where did you see that? Maybe I missed something yet again. This is unrelated to whether we ship a WAR if that's being conflated here. I rather doubt that embedded is on it's way out, although my memory isn't what it used to be. For starters, MapReduceIndexerTool uses it, so it gets regular exercise from that, and anything removing it would require some kind of replacement. How are you using it that you care? Wondering what alternatives exist... Best, Erick On Wed, Aug 5, 2015 at 9:09 AM, Robert Krüger krue...@lesspain.de wrote: Hi, I tried to upgrade my application from solr 4 to 5 and just now realized that embedded use of solr seems to be on the way out. Is that correct or is there a just new API to use for that? Thanks in advance, Robert
Re: Embedded Solr now deprecated?
On 8/5/2015 7:09 AM, Robert Krüger wrote: I tried to upgrade my application from solr 4 to 5 and just now realized that embedded use of solr seems to be on the way out. Is that correct or is there a just new API to use for that? Building on Erick's reply: I doubt that the embedded server is going away, and I do not recall seeing *anything* marking the entire class deprecated. The class still receives attention from devs -- this feature was released with 5.1.0: https://issues.apache.org/jira/browse/SOLR-7307 That said, we have discouraged users from deploying it in production for quite some time, even though it continues to exist and receive developer attention. Some of the reasons that I think users should avoid the embedded server: It doesn't support SolrCloud, you cannot make it fault-tolerant (redundant), and troubleshooting is harder because you cannot connect to it from outside of the source code where it is embedded. Deploying Solr as a network service offers much more capability than you can get when you embed it in your application. Chances are that you can easily replace EmbeddedSolrServer with one of the SolrClient classes and use a separate Solr deployment from your application. Thanks, Shawn
Re: Embedded Solr now deprecated?
I thought the Embedded server was good for a scenario where you wanted quickly to build a core with lots of documents locally. And then, move the core into production and swap it in. So you minimize the network traffic. Regards, Alex. Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter: http://www.solr-start.com/ On 5 August 2015 at 10:54, Shawn Heisey apa...@elyograg.org wrote: On 8/5/2015 7:09 AM, Robert Krüger wrote: I tried to upgrade my application from solr 4 to 5 and just now realized that embedded use of solr seems to be on the way out. Is that correct or is there a just new API to use for that? Building on Erick's reply: I doubt that the embedded server is going away, and I do not recall seeing *anything* marking the entire class deprecated. The class still receives attention from devs -- this feature was released with 5.1.0: https://issues.apache.org/jira/browse/SOLR-7307 That said, we have discouraged users from deploying it in production for quite some time, even though it continues to exist and receive developer attention. Some of the reasons that I think users should avoid the embedded server: It doesn't support SolrCloud, you cannot make it fault-tolerant (redundant), and troubleshooting is harder because you cannot connect to it from outside of the source code where it is embedded. Deploying Solr as a network service offers much more capability than you can get when you embed it in your application. Chances are that you can easily replace EmbeddedSolrServer with one of the SolrClient classes and use a separate Solr deployment from your application. Thanks, Shawn
Re: Embedded Solr now deprecated?
I just saw lots of deprecation warnings in my current code and a method that was removed, which is why I asked. Regarding the use case, I am embedding it with a desktop application just as others use java-based no-sql or rdbms engines and that makes sense architecturally in my case and is just simpler than deploying a separate little tomcat instance. API-wise, I know it is the same and it would be doable to do it that way. The embedded option is just the logical and simpler choice in terms of delivery, packaging, installation, automated testing in this case. The network option just doesn't add anything here apart from overhead (probably negligible in our case) and complexity. So our use is in production but in a desktop way, not what people normally think about when they hear production use. Thanks everyone for the quick feedback! I am very relieved to hear it is not on its way out and I will look at the api changes more closely and try to get our application running on 5.2.1. Best regards, Robert On Wed, Aug 5, 2015 at 4:34 PM, Erick Erickson erickerick...@gmail.com wrote: Where did you see that? Maybe I missed something yet again. This is unrelated to whether we ship a WAR if that's being conflated here. I rather doubt that embedded is on it's way out, although my memory isn't what it used to be. For starters, MapReduceIndexerTool uses it, so it gets regular exercise from that, and anything removing it would require some kind of replacement. How are you using it that you care? Wondering what alternatives exist... Best, Erick On Wed, Aug 5, 2015 at 9:09 AM, Robert Krüger krue...@lesspain.de wrote: Hi, I tried to upgrade my application from solr 4 to 5 and just now realized that embedded use of solr seems to be on the way out. Is that correct or is there a just new API to use for that? Thanks in advance, Robert -- Robert Krüger Managing Partner Lesspain GmbH Co. KG www.lesspain-software.com
Re: Embedded Solr now deprecated?
Hmmm, you may want to investigate the new no-war solution. Solr runs as a service with start/stop scripts. Currently it uses an underlying Jetty container, but that will (probably) eventually change and is pretty much considered an implementation detail. Not quite sure whether it'd be easier or harder for you though. Best, Erick On Wed, Aug 5, 2015 at 1:08 PM, Robert Krüger krue...@lesspain.de wrote: I just saw lots of deprecation warnings in my current code and a method that was removed, which is why I asked. Regarding the use case, I am embedding it with a desktop application just as others use java-based no-sql or rdbms engines and that makes sense architecturally in my case and is just simpler than deploying a separate little tomcat instance. API-wise, I know it is the same and it would be doable to do it that way. The embedded option is just the logical and simpler choice in terms of delivery, packaging, installation, automated testing in this case. The network option just doesn't add anything here apart from overhead (probably negligible in our case) and complexity. So our use is in production but in a desktop way, not what people normally think about when they hear production use. Thanks everyone for the quick feedback! I am very relieved to hear it is not on its way out and I will look at the api changes more closely and try to get our application running on 5.2.1. Best regards, Robert On Wed, Aug 5, 2015 at 4:34 PM, Erick Erickson erickerick...@gmail.com wrote: Where did you see that? Maybe I missed something yet again. This is unrelated to whether we ship a WAR if that's being conflated here. I rather doubt that embedded is on it's way out, although my memory isn't what it used to be. For starters, MapReduceIndexerTool uses it, so it gets regular exercise from that, and anything removing it would require some kind of replacement. How are you using it that you care? Wondering what alternatives exist... Best, Erick On Wed, Aug 5, 2015 at 9:09 AM, Robert Krüger krue...@lesspain.de wrote: Hi, I tried to upgrade my application from solr 4 to 5 and just now realized that embedded use of solr seems to be on the way out. Is that correct or is there a just new API to use for that? Thanks in advance, Robert -- Robert Krüger Managing Partner Lesspain GmbH Co. KG www.lesspain-software.com
RE: Embedded Solr now deprecated?
Hi Shawn, We have a different use case than the ones you covered in your response to Robert (below), which I wanted to call out. We currently use the embedded server when building indexes as part of a Hadoop workflow. The results get copied to a production analytics server and swapped in on a daily basis. Writing to multiple embedded servers (one per reduce task) gives us maximum performance, and has proven to be a very reliable method for the daily rebuild of pre-aggregations we need for our analytics use case. Regards, -- Ken PS - I'm also currently looking at using embedded Solr as a state storage engine for Samza. From: Shawn Heisey Sent: August 5, 2015 7:54:07am PDT To: solr-user@lucene.apache.org Subject: Re: Embedded Solr now deprecated? On 8/5/2015 7:09 AM, Robert Krüger wrote: I tried to upgrade my application from solr 4 to 5 and just now realized that embedded use of solr seems to be on the way out. Is that correct or is there a just new API to use for that? Building on Erick's reply: I doubt that the embedded server is going away, and I do not recall seeing *anything* marking the entire class deprecated. The class still receives attention from devs -- this feature was released with 5.1.0: https://issues.apache.org/jira/browse/SOLR-7307 That said, we have discouraged users from deploying it in production for quite some time, even though it continues to exist and receive developer attention. Some of the reasons that I think users should avoid the embedded server: It doesn't support SolrCloud, you cannot make it fault-tolerant (redundant), and troubleshooting is harder because you cannot connect to it from outside of the source code where it is embedded. Deploying Solr as a network service offers much more capability than you can get when you embed it in your application. Chances are that you can easily replace EmbeddedSolrServer with one of the SolrClient classes and use a separate Solr deployment from your application. Thanks, Shawn -- Ken Krugler +1 530-210-6378 http://www.scaleunlimited.com custom big data solutions training Hadoop, Cascading, Cassandra Solr
Re: Embedded Solr, event for cores up-and-running?
On 4/29/2015 9:26 AM, Erick Erickson wrote: I'm not sure there _is_ a good way short of sending a query at it. Since great efforts are made to have the embedded Solr act just like an external version, there's not much in the way of back-doors that I know of. Do the calls which create the EmbeddedSolrServer block until they are done, or return immediately and do their work in the background? Based on the code in the following example, I think they probably block until they're done: https://gist.github.com/delip/6214406 My guess is based on the fact that the code which indexes documents is immediately after the EmbeddedSolrServer constructor, with no wait for X loop. If we assume this code works, which it probably does, then I believe it MUST block until finished. I've never used the embedded server. If that guess is right, then it would be very easy to know when it's ready to use -- just wait for the constructor to finish. Thanks, Shawn
Embedded Solr, event for cores up-and-running?
If I run Solr in ebmedded mode (which I shouldn't, I know ;) ) how do I know (event?) that the cores are up-and-running, i.e. all is initialized? Thx Clemens
Re: Embedded Solr, event for cores up-and-running?
I'm not sure there _is_ a good way short of sending a query at it. Since great efforts are made to have the embedded Solr act just like an external version, there's not much in the way of back-doors that I know of. Best, Erick On Wed, Apr 29, 2015 at 6:49 AM, Clemens Wyss DEV clemens...@mysign.ch wrote: If I run Solr in ebmedded mode (which I shouldn't, I know ;) ) how do I know (event?) that the cores are up-and-running, i.e. all is initialized? Thx Clemens
Re: Dependency Need to include for embedded solr.
On 2/27/2015 12:51 AM, Danesh Kuruppu wrote: I am doing some feasibility studies for moving directly to solr 5.0.0. One more thing, It is related to standalone server. How security handle in solr standalone server. lets say, I configured my application to use remote solr standalone server. 1. How I would enable secure communication between my application and solr server. 2. How solr server authenticate user. Solr itself does not contain any security mechanisms, because Solr does not own *any* of the network communication layers where such security must be implemented. It is not currently possible for Solr to implement any reasonable security mechanisms. Eventually (hopefully in the near future), Solr will be a completely standalone application that does not rely on a servlet container, and when that happens, it will be possible to implement security within Solr itself. Right now, configuring SSL is not very hard, and you can also enable authentication in the servlet container. It's my understanding that using certificate-based authentication works already, but if you configure basic authentication (username/password), you will find any kind of distributed searching (including SolrCloud) will not function correctly. This is because Solr does not currently have any mechanism to provide the username/password when communicating with another instance. Thanks, Shawn
Re: Dependency Need to include for embedded solr.
On 2/26/2015 11:41 PM, Danesh Kuruppu wrote: My application is a standalone application. I though of embedding solr server, so I can pack it inside my application. In solr 5.0.0, solr is no longer distributed as a war file. how I can find the war file from the distribution. I am glad to see that people are actually reading documentation that is included with the release. With 5.0.0 (and probably the next few 5.x releases), Solr actually still is a war file. You can find it in server/webapps in the binary download. There are two reason we are telling everyone it's not a war file: 1) We now have very capable scripts to start and stop Solr with optimal java options, so there's no longer any need to rely on scripts packaged with a servlet container. 2) In a future 5.x release, Solr actually will become a standalone application, not a war ... preparing users in advance is a good idea. I need some advanced features like synonyms search, stop words, wild card search etc. It would be great, if you can provide some references to get idea which dependencies need to add to get those features. If you don't want to simply add every dependency included in the war, then you can use the tried and true method for finding the minimum set of jars: Try to get it running. If it fails, look at the log and see which class it was unable to find. Add the relevant jar to the classpath and try again. Thanks, Shawn
Re: Dependency Need to include for embedded solr.
Thanks shawn, I am doing some feasibility studies for moving directly to solr 5.0.0. One more thing, It is related to standalone server. How security handle in solr standalone server. lets say, I configured my application to use remote solr standalone server. 1. How I would enable secure communication between my application and solr server. 2. How solr server authenticate user. Thanks Danesh On Fri, Feb 27, 2015 at 12:35 PM, Shawn Heisey apa...@elyograg.org wrote: On 2/26/2015 11:41 PM, Danesh Kuruppu wrote: My application is a standalone application. I though of embedding solr server, so I can pack it inside my application. In solr 5.0.0, solr is no longer distributed as a war file. how I can find the war file from the distribution. I am glad to see that people are actually reading documentation that is included with the release. With 5.0.0 (and probably the next few 5.x releases), Solr actually still is a war file. You can find it in server/webapps in the binary download. There are two reason we are telling everyone it's not a war file: 1) We now have very capable scripts to start and stop Solr with optimal java options, so there's no longer any need to rely on scripts packaged with a servlet container. 2) In a future 5.x release, Solr actually will become a standalone application, not a war ... preparing users in advance is a good idea. I need some advanced features like synonyms search, stop words, wild card search etc. It would be great, if you can provide some references to get idea which dependencies need to add to get those features. If you don't want to simply add every dependency included in the war, then you can use the tried and true method for finding the minimum set of jars: Try to get it running. If it fails, look at the log and see which class it was unable to find. Add the relevant jar to the classpath and try again. Thanks, Shawn
Dependency Need to include for embedded solr.
Hi all, I need to include embed solr server into my maven project. I am going to use latest solr 5.0.0. Need to know which dependencies I need to include in my project. As I understand, I need to have solr-core[1] and solr-solrj[2]. Do I need to include lucene dependency in my project. If so, which dependencies we need to include to enable all indexing capabilities. 1. http://mvnrepository.com/artifact/org.apache.solr/solr-core/5.0.0 2. http://mvnrepository.com/artifact/org.apache.solr/solr-solrj/5.0.0 Please help Thanks Danesh
Re: Dependency Need to include for embedded solr.
On 2/26/2015 10:07 PM, Danesh Kuruppu wrote: I need to include embed solr server into my maven project. I am going to use latest solr 5.0.0. Need to know which dependencies I need to include in my project. As I understand, I need to have solr-core[1] and solr-solrj[2]. Do I need to include lucene dependency in my project. If so, which dependencies we need to include to enable all indexing capabilities. 1. http://mvnrepository.com/artifact/org.apache.solr/solr-core/5.0.0 2. http://mvnrepository.com/artifact/org.apache.solr/solr-solrj/5.0.0 Using the embedded server may not be the best idea. A lot of Solr functionality is not available in the embedded server. You can't use SolrCloud, which is a relatively easy way to provide high availability. The legacy method for redundancy, master-slave replication, also does not work in the embedded server. The admin UI is not available. If you choose to go ahead with the embedded server ... for complete safety, you should probably extract the war file and copy all the jars from WEB-INF/lib. If you want to take a more minimalistic approach, I think these are the Lucene jars you will need for minimum functionality: lucene-analyzers-common-5.0.0.jar lucene-codecs-5.0.0.jar lucene-core-5.0.0.jar lucene-expressions-5.0.0.jar lucene-queries-5.0.0.jar lucene-queryparser-5.0.0.jar There are quite a few Lucene jars, and I'm not overly familiar with everything that Solr uses, so I might have left some out that would be required for very basic functionality. For more advanced functionality, additional Lucene jars will definitely be required. There are also third-party jars that are required, such as slf4j jars for logging. The codebase as a whole has dependencies on things like google guava, several apache commons jars, and other pieces ... I have no idea which of those can be left out when using the embedded server. I tried to find a definitive list of required jars, and was not able to locate one. Thanks, Shawn
Re: Dependency Need to include for embedded solr.
Thanks Shawn, My application is a standalone application. I though of embedding solr server, so I can pack it inside my application. In solr 5.0.0, solr is no longer distributed as a war file. how I can find the war file from the distribution. I need some advanced features like synonyms search, stop words, wild card search etc. It would be great, if you can provide some references to get idea which dependencies need to add to get those features. Thanks Danesh On Fri, Feb 27, 2015 at 11:32 AM, Shawn Heisey apa...@elyograg.org wrote: On 2/26/2015 10:07 PM, Danesh Kuruppu wrote: I need to include embed solr server into my maven project. I am going to use latest solr 5.0.0. Need to know which dependencies I need to include in my project. As I understand, I need to have solr-core[1] and solr-solrj[2]. Do I need to include lucene dependency in my project. If so, which dependencies we need to include to enable all indexing capabilities. 1. http://mvnrepository.com/artifact/org.apache.solr/solr-core/5.0.0 2. http://mvnrepository.com/artifact/org.apache.solr/solr-solrj/5.0.0 Using the embedded server may not be the best idea. A lot of Solr functionality is not available in the embedded server. You can't use SolrCloud, which is a relatively easy way to provide high availability. The legacy method for redundancy, master-slave replication, also does not work in the embedded server. The admin UI is not available. If you choose to go ahead with the embedded server ... for complete safety, you should probably extract the war file and copy all the jars from WEB-INF/lib. If you want to take a more minimalistic approach, I think these are the Lucene jars you will need for minimum functionality: lucene-analyzers-common-5.0.0.jar lucene-codecs-5.0.0.jar lucene-core-5.0.0.jar lucene-expressions-5.0.0.jar lucene-queries-5.0.0.jar lucene-queryparser-5.0.0.jar There are quite a few Lucene jars, and I'm not overly familiar with everything that Solr uses, so I might have left some out that would be required for very basic functionality. For more advanced functionality, additional Lucene jars will definitely be required. There are also third-party jars that are required, such as slf4j jars for logging. The codebase as a whole has dependencies on things like google guava, several apache commons jars, and other pieces ... I have no idea which of those can be left out when using the embedded server. I tried to find a definitive list of required jars, and was not able to locate one. Thanks, Shawn
Re: Set up embedded Solr container and cores programmatically to read their configs from the classpath
Hi Robert, I don't think this is possible at the moment, but I hope to get https://issues.apache.org/jira/browse/SOLR-4478 in for Lucene/Solr 4.7, which should allow you to inject your own SolrResourceLoader implementation for core creation (it sounds as though you want to wrap the core's loader in a ClasspathResourceLoader). You could try applying that patch to your setup and see if that helps you out. Alan Woodward www.flax.co.uk On 11 Feb 2014, at 10:41, Robert Krüger wrote: Hi, I have an application with an embedded Solr instance (and I want to keep it embedded) and so far I have been setting up my Solr installation programmatically using folder paths to specify where the specific container or core configs are. I have used the CoreContainer methods createAndLoad and create using File arguments and this works fine. However, now I want to change this so that all configuration files are loaded from certain locations using the classloader but I have not been able to get this to work. E.g. I want to have my solr config located in the classpath at my/base/package/solr/conf and the core configs at my/base/package/solr/cores/core1/conf, my/base/package/solr/cores/core2/conf etc.. Is this possible at all? Looking through the source code it seems that specifying classpath resources in such a qualified way is not supported but I may be wrong. I could get this to work for the container by supplying my own implementation of SolrResourceLoader that allows a base path to be specified for the resources to be loaded (I first thought that would happen already when specifying instanceDir accordingly but looking at the code it does not. for resources loaded through the classloader, instanceDir is not prepended). However then I am stuck with the loading of the cores' resources as the respective code (see org.apache.solr.core.CoreContainer#createFromLocal) instantiates a SolResourceLoader internally. Thanks for any help with this (be it a clarification that it is not possible). Robert
Set up embedded Solr container and cores programmatically to read their configs from the classpath
Hi, I have an application with an embedded Solr instance (and I want to keep it embedded) and so far I have been setting up my Solr installation programmatically using folder paths to specify where the specific container or core configs are. I have used the CoreContainer methods createAndLoad and create using File arguments and this works fine. However, now I want to change this so that all configuration files are loaded from certain locations using the classloader but I have not been able to get this to work. E.g. I want to have my solr config located in the classpath at my/base/package/solr/conf and the core configs at my/base/package/solr/cores/core1/conf, my/base/package/solr/cores/core2/conf etc.. Is this possible at all? Looking through the source code it seems that specifying classpath resources in such a qualified way is not supported but I may be wrong. I could get this to work for the container by supplying my own implementation of SolrResourceLoader that allows a base path to be specified for the resources to be loaded (I first thought that would happen already when specifying instanceDir accordingly but looking at the code it does not. for resources loaded through the classloader, instanceDir is not prepended). However then I am stuck with the loading of the cores' resources as the respective code (see org.apache.solr.core.CoreContainer#createFromLocal) instantiates a SolResourceLoader internally. Thanks for any help with this (be it a clarification that it is not possible). Robert
Re: solrj indexing using embedded solr is slow
First, how much slower? 2x? 10x? 1.1x? When using embedded, you're doing all the work you were doing on two machines on a single machine, so my first question would be how is your CPU performaing? Is it maxed? Best Erick On Thu, Jun 27, 2013 at 1:59 PM, Learner bbar...@gmail.com wrote: Shawn, Thanks a lot for your reply. I have pasted my entire code below, it would be great if you can let me know if I am doing anything wrong in terms of running the code in multithreaded environment. http://pastebin.com/WRLn3yWn -- View this message in context: http://lucene.472066.n3.nabble.com/solrj-indexing-using-embedded-solr-is-slow-tp4073636p4073711.html Sent from the Solr - User mailing list archive at Nabble.com.
solrj indexing using embedded solr is slow
I was using ConcurrentUpdateSOLR for indexing documents to Solr. Later I had a need to do portable indexing hence started using Embedded solr server. I created a multithreaded program to create /submit the documents in batch of 100 to Embedded SOLR server (running inside Solrj indexing process) but for some reason it takes more time to index the data when compared with ConcurrentUpdateSOLR server(CUSS). I was under assumption that embedded server would take less time compared to http update (made when using CUSS) but not sure why it takes more time... Is there a way to speed up the indexing when using Embedded solr serveretc..(something like specifying thread and queue size similar to CUSS)? -- View this message in context: http://lucene.472066.n3.nabble.com/solrj-indexing-using-embedded-solr-is-slow-tp4073636.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: solrj indexing using embedded solr is slow
On 6/27/2013 9:19 AM, Learner wrote: I was using ConcurrentUpdateSOLR for indexing documents to Solr. Later I had a need to do portable indexing hence started using Embedded solr server. I created a multithreaded program to create /submit the documents in batch of 100 to Embedded SOLR server (running inside Solrj indexing process) but for some reason it takes more time to index the data when compared with ConcurrentUpdateSOLR server(CUSS). I was under assumption that embedded server would take less time compared to http update (made when using CUSS) but not sure why it takes more time... Is there a way to speed up the indexing when using Embedded solr serveretc..(something like specifying thread and queue size similar to CUSS)? A lot more time has been spent optimizing the traditional Solr server model than the embedded version. If you want the same performance from Embedded that you get from Concurrent, you'll need to use that object in multiple threads that you create yourself. The Concurrent object handles all that threading for you, but due to its nature, Embedded can't. You say that your program is multithreaded, so I really don't know what's going on here. An FYI that on something that might have escaped your awareness: CUSS swallows exceptions - it will never inform the calling application about errors that occur, unless you override its handleError method in some way, and I don't know what is required to make it do that. This is part of why CUSS is so fast - it returns to the calling application *immediately*, no matter what actually happens in the background while talking to the server. Thanks, Shawn
Re: solrj indexing using embedded solr is slow
Shawn, Thanks a lot for your reply. I have pasted my entire code below, it would be great if you can let me know if I am doing anything wrong in terms of running the code in multithreaded environment. http://pastebin.com/WRLn3yWn -- View this message in context: http://lucene.472066.n3.nabble.com/solrj-indexing-using-embedded-solr-is-slow-tp4073636p4073711.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: discovery-based core enumeration with embedded solr
A, good catch! Coincidentally yesterday while in the midst of looking at some other JIRAs, I noticed that some pages on the Wiki said 4.2 and changed what I ran across to 4.3. I originally started the Wikis when I though I would go fast enough to get it into 4.2, sorry for the confusion! If the Wiki page you were looking at still says 4.2, please either edit it directly or send the link to me and I'll change it. So you're right, this isn't in 4.2 at all so the error above isn't in the new code cause there is no relevant new code to be wrong G. If you could pull a 4.x (or trunk) copy and try that would be great. Nightly builds are linked from here: http://wiki.apache.org/solr/NightlyBuilds I should also add that I'm eager for people to kick the tires on this so we can harden the whole thing before 4.3 actually gets cut. Best Erick On Fri, Mar 15, 2013 at 11:23 AM, Michael Sokolov msoko...@safaribooksonline.com wrote: Erick, before I do that - which I'll be happy to - I just want to make sure I'm testing the right thing. The wiki seems to indicate this is a 4.2+ feature, but the ticket marks it as fixed in 4.3. Maybe just a document bug? -Mike On 3/14/13 9:44 PM, Erick Erickson wrote: H, could you raise a JIRA and assign it to me? Please be sure and emphasize that it's embedded because I'm pretty sure this is fine for the regular case. But I have to admit that the embedded case completely slipped under the radar. Even better if you could make a test case, but that might not be straightforward... Thanks, Erick On Wed, Mar 13, 2013 at 5:28 PM, Michael Sokolov msoko...@safaribooksonline.com** wrote: Has the new core enumeration strategy been implemented in the CoreContainer.Initializer.initialize() code path? It doesn't seem like it has. I get this exception: Caused by: org.apache.solr.common.SolrException: Could not load config for solrconfig.xml at org.apache.solr.core.CoreContainer.createFromLocal( CoreContainer.java:991) at org.apache.solr.core.CoreContainer.create(** CoreContainer.java:1051) ... 10 more Caused by: java.io.IOException: Can't find resource 'solrconfig.xml' in classpath or 'solr-multi/collection1/conf/', cwd=/proj/lux at org.apache.solr.core.SolrResourceLoader.openResource(** SolrResourceLoader.java:318) at org.apache.solr.core.SolrResourceLoader.openConfig( SolrResourceLoader.java:283) at org.apache.solr.core.Config.init(Config.java:103) at org.apache.solr.core.Config.init(Config.java:73) at org.apache.solr.core.SolrConfig.init(SolrConfig.*** *java:117) at org.apache.solr.core.CoreContainer.createFromLocal( CoreContainer.java:989) ... 11 more even though I have a solr.properties file in solr-multi (which is my solr.home), and core.properties in some subdirectories of that -- Michael Sokolov Senior Architect Safari Books Online
Re: discovery-based core enumeration with embedded solr
On 3/16/2013 9:52 AM, Erick Erickson wrote: A, good catch! Coincidentally yesterday while in the midst of looking at some other JIRAs, I noticed that some pages on the Wiki said 4.2 and changed what I ran across to 4.3. I originally started the Wikis when I though I would go fast enough to get it into 4.2, sorry for the confusion! If the Wiki page you were looking at still says 4.2, please either edit it directly or send the link to me and I'll change it. You've updated the confused info I saw, now. Thanks that clears it up for me. So you're right, this isn't in 4.2 at all so the error above isn't in the new code cause there is no relevant new code to be wrong G. If you could pull a 4.x (or trunk) copy and try that would be great. Nightly builds are linked from here: http://wiki.apache.org/solr/NightlyBuilds I should also add that I'm eager for people to kick the tires on this so we can harden the whole thing before 4.3 actually gets cut. I'll let you know if I run into any issues there - I'm not sure if I'll get to testing this before you release, but I'll try! -Mike
Re: discovery-based core enumeration with embedded solr
Erick, before I do that - which I'll be happy to - I just want to make sure I'm testing the right thing. The wiki seems to indicate this is a 4.2+ feature, but the ticket marks it as fixed in 4.3. Maybe just a document bug? -Mike On 3/14/13 9:44 PM, Erick Erickson wrote: H, could you raise a JIRA and assign it to me? Please be sure and emphasize that it's embedded because I'm pretty sure this is fine for the regular case. But I have to admit that the embedded case completely slipped under the radar. Even better if you could make a test case, but that might not be straightforward... Thanks, Erick On Wed, Mar 13, 2013 at 5:28 PM, Michael Sokolov msoko...@safaribooksonline.com wrote: Has the new core enumeration strategy been implemented in the CoreContainer.Initializer.**initialize() code path? It doesn't seem like it has. I get this exception: Caused by: org.apache.solr.common.**SolrException: Could not load config for solrconfig.xml at org.apache.solr.core.**CoreContainer.createFromLocal(** CoreContainer.java:991) at org.apache.solr.core.**CoreContainer.create(** CoreContainer.java:1051) ... 10 more Caused by: java.io.IOException: Can't find resource 'solrconfig.xml' in classpath or 'solr-multi/collection1/conf/'**, cwd=/proj/lux at org.apache.solr.core.**SolrResourceLoader.**openResource(** SolrResourceLoader.java:318) at org.apache.solr.core.**SolrResourceLoader.openConfig(** SolrResourceLoader.java:283) at org.apache.solr.core.Config.**init(Config.java:103) at org.apache.solr.core.Config.**init(Config.java:73) at org.apache.solr.core.**SolrConfig.init(SolrConfig.**java:117) at org.apache.solr.core.**CoreContainer.createFromLocal(** CoreContainer.java:989) ... 11 more even though I have a solr.properties file in solr-multi (which is my solr.home), and core.properties in some subdirectories of that -- Michael Sokolov Senior Architect Safari Books Online
Re: Embedded Solr
);
if((i + 1) % commitNum == 0) {
long t = System.currentTimeMillis();
solrServer.add(docs);
solrServer.commit();
System.err.println(computerName + +
Thread.currentThread().getId()
+ : + (System.currentTimeMillis() - t));
totalTime = System.currentTimeMillis()
- t + totalTime;
docs.clear();
}
}
if(docs.size() 0) {
long t = System.currentTimeMillis();
solrServer.add(docs);
solrServer.commit();
System.err.println(computerName + +
Thread.currentThread().getId() +
: + (System.currentTimeMillis() - t));
totalTime = System.currentTimeMillis() - t +
totalTime;
}
} catch(MalformedURLException e) {
e.printStackTrace();
} catch(Exception e) {
e.printStackTrace();
} finally {
}
System.out.println(end time: +
Thread.currentThread().getId() + \t
+ (System.currentTimeMillis() - beginTime));
System.out.println(end time total: +
Thread.currentThread().getId() +
t \t + totalTime);
solrServer.shutdown();
}
--
View this message in context:
http://lucene.472066.n3.nabble.com/Embedded-Solr-tp481608p4047562.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: discovery-based core enumeration with embedded solr
H, could you raise a JIRA and assign it to me? Please be sure and emphasize that it's embedded because I'm pretty sure this is fine for the regular case. But I have to admit that the embedded case completely slipped under the radar. Even better if you could make a test case, but that might not be straightforward... Thanks, Erick On Wed, Mar 13, 2013 at 5:28 PM, Michael Sokolov msoko...@safaribooksonline.com wrote: Has the new core enumeration strategy been implemented in the CoreContainer.Initializer.**initialize() code path? It doesn't seem like it has. I get this exception: Caused by: org.apache.solr.common.**SolrException: Could not load config for solrconfig.xml at org.apache.solr.core.**CoreContainer.createFromLocal(** CoreContainer.java:991) at org.apache.solr.core.**CoreContainer.create(** CoreContainer.java:1051) ... 10 more Caused by: java.io.IOException: Can't find resource 'solrconfig.xml' in classpath or 'solr-multi/collection1/conf/'**, cwd=/proj/lux at org.apache.solr.core.**SolrResourceLoader.**openResource(** SolrResourceLoader.java:318) at org.apache.solr.core.**SolrResourceLoader.openConfig(** SolrResourceLoader.java:283) at org.apache.solr.core.Config.**init(Config.java:103) at org.apache.solr.core.Config.**init(Config.java:73) at org.apache.solr.core.**SolrConfig.init(SolrConfig.**java:117) at org.apache.solr.core.**CoreContainer.createFromLocal(** CoreContainer.java:989) ... 11 more even though I have a solr.properties file in solr-multi (which is my solr.home), and core.properties in some subdirectories of that -- Michael Sokolov Senior Architect Safari Books Online
discovery-based core enumeration with embedded solr
Has the new core enumeration strategy been implemented in the CoreContainer.Initializer.initialize() code path? It doesn't seem like it has. I get this exception: Caused by: org.apache.solr.common.SolrException: Could not load config for solrconfig.xml at org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:991) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1051) ... 10 more Caused by: java.io.IOException: Can't find resource 'solrconfig.xml' in classpath or 'solr-multi/collection1/conf/', cwd=/proj/lux at org.apache.solr.core.SolrResourceLoader.openResource(SolrResourceLoader.java:318) at org.apache.solr.core.SolrResourceLoader.openConfig(SolrResourceLoader.java:283) at org.apache.solr.core.Config.init(Config.java:103) at org.apache.solr.core.Config.init(Config.java:73) at org.apache.solr.core.SolrConfig.init(SolrConfig.java:117) at org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:989) ... 11 more even though I have a solr.properties file in solr-multi (which is my solr.home), and core.properties in some subdirectories of that -- Michael Sokolov Senior Architect Safari Books Online
Re: embedded solr populating field of type LatLonType
On Tue, Apr 24, 2012 at 4:05 PM, Jason Cunning jcunn...@ucar.edu wrote: My question is, what is the AppropriateJavaType for populating a solr field of type LatLonType? A String with both the lat and lon separated by a comma. Example: 12.34,56.78 -Yonik lucenerevolution.com - Lucene/Solr Open Source Search Conference. Boston May 7-10
embedded solr populating field of type LatLonType
Hi,
I have a question concerning the spatial field type LatLonType and populating
it via an embedded solr server in java.
So far I've only ever had to index simple types like boolean, float, and
string. This is the first complex type. So I'd like to use the following field
definition for example in my schema:
field name=coordinate type=LatLonType indexed=true stored=false
multiValued=false/
And then I'd like to populate this field in java as in the following psuedo
code:
public SolrInputDocument populate(AppropriateJavaType coordinate) {
SolrInputField inputField = new SolrInputField(coordinate);
inputField.addValue(coordinate, 1.0F);
SolrInputDocument inputDocument = new SolrInputDocument();
inputDocument.put(coordinate, inputField);
return inputDocument;
}
My question is, what is the AppropriateJavaType for populating a solr field of
type LatLonType?
Thank you for your time.
How to connect embedded solr with each other by sharding
I have been using sharding with multiple basic solr server for clustering. I also used one embedded solr server (Solrj Java API) with many basic solr servers and connecting them by sharding as embedded solr server is the caller of them. I used the code line below for this purpose. SolrQuery query = new SolrQuery(); query.set(shards, solr1URL,solr2URL,...); Now, I have many embedded solr servers running on different computers and they are unaware of each others. I want to communicate them with each other by sharding. Is it possible? if yes how? if not what are the other options that you can advice by using embedded solr servers? -- View this message in context: http://lucene.472066.n3.nabble.com/How-to-connect-embedded-solr-with-each-other-by-sharding-tp3753337p3753337.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Exceptions in Embedded Solr
I also got the same exception, anyone know how to solve this? -- View this message in context: http://lucene.472066.n3.nabble.com/Exceptions-in-Embedded-Solr-tp2007304p3057746.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Embedded Solr Optimize under Windows
: Thanks for the reply. I'm at home right now, or I'd try this myself, but is : the suggestion that two optimize() calls in a row would resolve the issue? it might ... I think the situations in which it happens have evolved a bit over the years as IndexWRiter has gotten smarter about knowing when it really needs to touch the disk to reduce IO. there's a relatively new explicit method (IndexWriter.deleteUnusedFiles) that can force this... https://issues.apache.org/jira/browse/LUCENE-2259 ...but it's only on trunk, and there isn't any user level hook for it in Solr yet (i opened SOLR-2532 to consider adding it) -Hoss
Re: Embedded Solr Optimize under Windows
Ahh, thanks. I might try a basic commit() then and see, although it's not a huge deal for me. It occurred to me that two optimize() calls would probably leave exactly the same problem behind. On 20 May 2011 09:52, Chris Hostetter hossman_luc...@fucit.org wrote: : Thanks for the reply. I'm at home right now, or I'd try this myself, but is : the suggestion that two optimize() calls in a row would resolve the issue? it might ... I think the situations in which it happens have evolved a bit over the years as IndexWRiter has gotten smarter about knowing when it really needs to touch the disk to reduce IO. there's a relatively new explicit method (IndexWriter.deleteUnusedFiles) that can force this... https://issues.apache.org/jira/browse/LUCENE-2259 ...but it's only on trunk, and there isn't any user level hook for it in Solr yet (i opened SOLR-2532 to consider adding it) -Hoss
Re: Embedded Solr Optimize under Windows
: http://code.google.com/p/solr-geonames/wiki/DeveloperInstall : It's worth noting that the build has also been run on Mac and Solaris now, : and the Solr index is about half the size. We suspect the optimize() call in : Embedded Solr is not working correctly under Windows. : : We've observed that Windows leaves lots of segments on disk and takes up : twice the volume as the other OSs. Perhaps file locking or something The problem isn't that optimize doesn't work on windows, the problem is that windows file semantics won't let files be deleted while there are open file handles -- so Lucene's Directory behavior is to leave the files on disk, and try to clean them up later. (on the next write, or next optimize call) -Hoss
Re: Embedded Solr Optimize under Windows
Thanks for the reply. I'm at home right now, or I'd try this myself, but is the suggestion that two optimize() calls in a row would resolve the issue? The process in question is a JVM devoted entirely to harvesting, calls optimize() then shuts down. The least processor intensive way of triggering this behaviour is desirable... perhaps a commit()? But I wouldn't have expected that to trigger a write. On 17 May 2011 10:20, Chris Hostetter hossman_luc...@fucit.org wrote: : http://code.google.com/p/solr-geonames/wiki/DeveloperInstall : It's worth noting that the build has also been run on Mac and Solaris now, : and the Solr index is about half the size. We suspect the optimize() call in : Embedded Solr is not working correctly under Windows. : : We've observed that Windows leaves lots of segments on disk and takes up : twice the volume as the other OSs. Perhaps file locking or something The problem isn't that optimize doesn't work on windows, the problem is that windows file semantics won't let files be deleted while there are open file handles -- so Lucene's Directory behavior is to leave the files on disk, and try to clean them up later. (on the next write, or next optimize call) -Hoss
Embedded Solr Optimize under Windows
Hi All, Just quick query of no particular importance to me, but we did observe this problem: http://code.google.com/p/solr-geonames/wiki/DeveloperInstall It's worth noting that the build has also been run on Mac and Solaris now, and the Solr index is about half the size. We suspect the optimize() call in Embedded Solr is not working correctly under Windows. We've observed that Windows leaves lots of segments on disk and takes up twice the volume as the other OSs. Perhaps file locking or something prevents the optimize() call from functioning. This wasn't particularly important to us since we don't run Windows for any prod systems. For that reason we haven't looked too closely, but thought it might be of interest to others... if we are even right of course :) Ta, Greg
RE: Embedded Solr constructor not returning
Hi Greg, I need the servlet API in my app for it to work, despite being command line. So adding this to the maven POM fixed everything: dependency groupIdjavax.servlet/groupId artifactIdservlet-api/artifactId version2.5/version /dependency Perhaps this dependency could be listed on the wiki? Alongside the sample code for using embedded solr? http://wiki.apache.org/solr/Solrj Sounds good. Please go ahead and make this change yourself. FYI, the Solr 3.1 POM has a servlet-api dependency, but the scope is provided, because the servlet container includes this dependency. When *you* are the container, you have to provide it. Steve
Re: Embedded Solr constructor not returning
Sounds good. Please go ahead and make this change yourself. Done. Ta, Greg On 6 April 2011 22:52, Steven A Rowe sar...@syr.edu wrote: Hi Greg, I need the servlet API in my app for it to work, despite being command line. So adding this to the maven POM fixed everything: dependency groupIdjavax.servlet/groupId artifactIdservlet-api/artifactId version2.5/version /dependency Perhaps this dependency could be listed on the wiki? Alongside the sample code for using embedded solr? http://wiki.apache.org/solr/Solrj Sounds good. Please go ahead and make this change yourself. FYI, the Solr 3.1 POM has a servlet-api dependency, but the scope is provided, because the servlet container includes this dependency. When *you* are the container, you have to provide it. Steve
Embedded Solr constructor not returning
Hi All,
I'm hoping this is a reasonably trivial issue, but it's frustrating me to no
end. I'm putting together a tiny command line app to write data into an
index. It has no web based Solr running against it; the index will be moved
at a later time to have a proper server instance start for responding to
queries. My problem however is I seem to have stalled on instantiating the
embedded server:
private SolrServer startSolr(String home) throws Exception {
try {
System.setProperty(solr.solr.home, home);
CoreContainer.Initializer initializer = new
CoreContainer.Initializer();
solrCore = initializer.initialize();
return new EmbeddedSolrServer(solrCore, );
} catch(Exception ex) {
log.error(\n===\nFailed to start Solr server\n);
throw ex;
}
}
The constructor for the embedded server just never comes back. I've seen
three or four different ways of starting the server with varying levels of
complexity, and they all APPEAR to work, but still do not return. STDOUT
show the output I have largely come to expect from watching Solr start
'correctly':
===
Starting Solr:
JNDI not configured for solr (NoInitialContextEx)
using system property solr.solr.home: C:\test\harvester\solr
looking for solr.xml: C:\test\harvester\solr\solr.xml
Solr home set to 'C:\test\harvester\solr\'
Loaded SolrConfig: solrconfig.xml
Opening new SolrCore at C:\test\harvester\solr\,
dataDir=C:\tf2\geonames\harvester\solr\.\data\
Reading Solr Schema
Schema name=test
created string: org.apache.solr.schema.StrField
created date: org.apache.solr.schema.TrieDateField
created sint: org.apache.solr.schema.SortableIntField
created sfloat: org.apache.solr.schema.SortableFloatField
created null: org.apache.solr.analysis.WhitespaceTokenizerFactory
created null: org.apache.solr.analysis.LowerCaseFilterFactory
created null: org.apache.solr.analysis.WhitespaceTokenizerFactory
created null: org.apache.solr.analysis.LowerCaseFilterFactory
created text: org.apache.solr.schema.TextField
default search field is basic_name
query parser default operator is AND
unique key field: id
No JMX servers found, not exposing Solr information with JMX.
created /update: solr.XmlUpdateRequestHandler
adding lazy requestHandler: solr.CSVRequestHandler
created /update/csv: solr.CSVRequestHandler
Opening Searcher@11b86c7 main
AutoCommit: disabled
registering core:
[] Registered new searcher Searcher@11b86c7 main
Terminate batch job (Y/N)? y
At this stage I'm grasping at straws. It appears as though the embedded
instance is behaving like a proper server, waiting for a request or
something. I've scrubbed the solrconfig.xml (from from the Solr example
download) file back to remove most entries, but perhaps I'm using the
incorrect handlers/listeners for an embedded server?
I'm a tad confused though, because every other time I've done this
(admittedly in a servlet, not a command line app) the constructor simply
returns straight away and execution of my app code continues.
Any advice or suggestions would be greatly appreciated.
Ta,
Greg
Re: Embedded Solr constructor not returning
Hmmm, after being stuck on this for hours, I find the answer myself
15minutes after asking for help... as usual. :)
For anyone interested, and no doubt this will not be a revelation for some,
I need the servlet API in my app for it to work, despite being command line.
So adding this to the maven POM fixed everything:
dependency
groupIdjavax.servlet/groupId
artifactIdservlet-api/artifactId
version2.5/version
/dependency
Perhaps this dependency could be listed on the wiki? Alongside the sample
code for using embedded solr?
http://wiki.apache.org/solr/Solrj
Logback is passing along all of my logging but I suspect I'd have to add
some Solr logging config before it would tell me this itself. I only
stumbled on it by accident:
http://osdir.com/ml/solr-user.lucene.apache.org/2009-11/msg00831.html
On 6 April 2011 14:48, Greg Pendlebury greg.pendleb...@gmail.com wrote:
Hi All,
I'm hoping this is a reasonably trivial issue, but it's frustrating me to
no end. I'm putting together a tiny command line app to write data into an
index. It has no web based Solr running against it; the index will be moved
at a later time to have a proper server instance start for responding to
queries. My problem however is I seem to have stalled on instantiating the
embedded server:
private SolrServer startSolr(String home) throws Exception {
try {
System.setProperty(solr.solr.home, home);
CoreContainer.Initializer initializer = new
CoreContainer.Initializer();
solrCore = initializer.initialize();
return new EmbeddedSolrServer(solrCore, );
} catch(Exception ex) {
log.error(\n===\nFailed to start Solr server\n);
throw ex;
}
}
The constructor for the embedded server just never comes back. I've seen
three or four different ways of starting the server with varying levels of
complexity, and they all APPEAR to work, but still do not return. STDOUT
show the output I have largely come to expect from watching Solr start
'correctly':
===
Starting Solr:
JNDI not configured for solr (NoInitialContextEx)
using system property solr.solr.home: C:\test\harvester\solr
looking for solr.xml: C:\test\harvester\solr\solr.xml
Solr home set to 'C:\test\harvester\solr\'
Loaded SolrConfig: solrconfig.xml
Opening new SolrCore at C:\test\harvester\solr\,
dataDir=C:\tf2\geonames\harvester\solr\.\data\
Reading Solr Schema
Schema name=test
created string: org.apache.solr.schema.StrField
created date: org.apache.solr.schema.TrieDateField
created sint: org.apache.solr.schema.SortableIntField
created sfloat: org.apache.solr.schema.SortableFloatField
created null: org.apache.solr.analysis.WhitespaceTokenizerFactory
created null: org.apache.solr.analysis.LowerCaseFilterFactory
created null: org.apache.solr.analysis.WhitespaceTokenizerFactory
created null: org.apache.solr.analysis.LowerCaseFilterFactory
created text: org.apache.solr.schema.TextField
default search field is basic_name
query parser default operator is AND
unique key field: id
No JMX servers found, not exposing Solr information with JMX.
created /update: solr.XmlUpdateRequestHandler
adding lazy requestHandler: solr.CSVRequestHandler
created /update/csv: solr.CSVRequestHandler
Opening Searcher@11b86c7 main
AutoCommit: disabled
registering core:
[] Registered new searcher Searcher@11b86c7 main
Terminate batch job (Y/N)? y
At this stage I'm grasping at straws. It appears as though the embedded
instance is behaving like a proper server, waiting for a request or
something. I've scrubbed the solrconfig.xml (from from the Solr example
download) file back to remove most entries, but perhaps I'm using the
incorrect handlers/listeners for an embedded server?
I'm a tad confused though, because every other time I've done this
(admittedly in a servlet, not a command line app) the constructor simply
returns straight away and execution of my app code continues.
Any advice or suggestions would be greatly appreciated.
Ta,
Greg
Re: Embedded Solr
Greg, I guess the question is related to the app-store's ban of java. If you follow the java-...@lists.apple.com mailing list you'll see that a full JVM is a problem but that skimmed, self-installed JVMs have been ok at least once. SOLR is under the family of things that do not need a gui so you do a little chance to skim-down efficiently into something that doesn't rely on a pre-installed jvm. I believe the results' of such an experiment interest this list (and the list above). paul Le 22 mars 2011 à 00:53, Bill Bell a écrit : Yes it needs java to run Bill Bell Sent from mobile On Mar 21, 2011, at 2:30 PM, Greg Georges greg.geor...@biztree.com wrote: Hello all, I am using Solr in a Java architecture right now, and the results are great. The app development team has asked me if it is possible to embed Solr, but the request is to embed it into a C++ app and mac app using objective C. I do not have much knowledge on embedded Solr. Does it need a JVM? Is what they are asking me possible and are there any resources for it? Thanks Greg
Embedded Solr
Hello all, I am using Solr in a Java architecture right now, and the results are great. The app development team has asked me if it is possible to embed Solr, but the request is to embed it into a C++ app and mac app using objective C. I do not have much knowledge on embedded Solr. Does it need a JVM? Is what they are asking me possible and are there any resources for it? Thanks Greg
Re: Embedded Solr
Yes it needs java to run Bill Bell Sent from mobile On Mar 21, 2011, at 2:30 PM, Greg Georges greg.geor...@biztree.com wrote: Hello all, I am using Solr in a Java architecture right now, and the results are great. The app development team has asked me if it is possible to embed Solr, but the request is to embed it into a C++ app and mac app using objective C. I do not have much knowledge on embedded Solr. Does it need a JVM? Is what they are asking me possible and are there any resources for it? Thanks Greg
Re: Special Circumstances for embedded Solr
Can you please show me how an http implementation of solrj querying can be converted to one for embedded solr with the help of an example? -- View this message in context: http://lucene.472066.n3.nabble.com/Special-Circumstances-for-embedded-Solr-tp833409p2566768.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Special Circumstances for embedded Solr
On 02/24/2011 12:16 PM, Devangini wrote:
Can you please show me how an http implementation of solrj querying can be
converted to one for embedded solr with the help of an example?
Hi, heres an example that almost compiles. You should be able to get
going with this.
T
class EmbeddedSolrExample {
public static void main (String[] args) {
setupSolrContainer();
addDocument();
SolrDocumentList getResults(QueryResponse response) {
if (response.getStatus() != 0) {
return new SolrDocumentList();
}
return response.getResults();
}
private void addDocument() throws IncompleteDocumentException,
SolrServerException, IOException {
SolrInputDocument res = new SolrInputDocument();
res.setField(body, test);
res.setField(id, 12);
UpdateResponse s = server.add(res);
assertEquals((int) s.getStatus(), 0 );
server.commit();
SolrDocumentList res = getResults(search(test));
System.out.println(I got + res.size() + documents);
}
private void setupSolrContainer() throws
ParserConfigurationException, IOException, SAXException,
IncompleteDocumentException, SolrServerException {
File home = new File(/tmp/solr);
File f = new File(home, solr.xml);
CoreContainer container = new CoreContainer();
container.load(/tmp/solr, f);
server = new EmbeddedSolrServer(container, model);
addDocument();
}
QueryResponse search(String words) throws SolrServerException {
SolrQuery query = new SolrQuery();
query.addField(id).addField(body).addField(score);
query.setTimeAllowed(1000);
query.setRows(50);
query.set(q, words);
query.setSortField(timestamp, ORDER.desc); // sorter på dato
return server.query(query);
}
--
Regards / Med vennlig hilsen
Tarjei Huse
Mobil: 920 63 413
embedded solr and tomcat
I'm considering running an embedded instance of Solr in Tomcat (Amazon's beanstalk). Has anyone done this before? I'd be very interested in how I can instantiate Embedded solr in Tomcat. Do I need a resource loader to instantiate? If so, how? Thanks, Matt
Re: Exceptions in Embedded Solr
I experienced this on an EmbeddedSolrServer which was running behind a
tomcat process. After restarting the tomcat process 2-3 times (implying this
also recreates the SolrServer every time as well) this issue went away but I
don't know why it ever started. It looked like the searcher shutdown was not
clean the previous time and I believe that could have to do with it.
Tony
On Sat, Dec 4, 2010 at 11:44 PM, Tharindu Mathew mcclou...@gmail.comwrote:
Any help on this?
On Thu, Dec 2, 2010 at 7:51 PM, Tharindu Mathew mcclou...@gmail.com
wrote:
Hi everyone,
I get the exception below when using Embedded Solr suddenly. If I
delete the Solr index it goes back to normal, but it obviously has to
start indexing from scratch. Any idea what the cause of this is?
java.lang.RuntimeException: java.io.FileNotFoundException:
/home/evanthika/WSO2/CARBON/GREG/3.6.0/23-11-2010/normal/wso2greg-3.6.0/solr/data/index/segments_2
(No such file or directory)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1068)
at org.apache.solr.core.SolrCore.init(SolrCore.java:579)
at
org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:137)
at
org.wso2.carbon.registry.indexing.solr.SolrClient.init(SolrClient.java:103)
at
org.wso2.carbon.registry.indexing.solr.SolrClient.getInstance(SolrClient.java:115)
... 44 more
Caused by: java.io.FileNotFoundException:
/home/evanthika/WSO2/CARBON/GREG/3.6.0/23-11-2010/normal/wso2greg-3.6.0/solr/data/index/segments_2
(No such file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.init(RandomAccessFile.java:212)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.init(SimpleFSDirectory.java:78)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.init(SimpleFSDirectory.java:108)
at
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.init(NIOFSDirectory.java:94)
at
org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:70)
at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:691)
at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:236)
at
org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:72)
at
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:683)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:69)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:476)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:403)
at
org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:38)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1057)
... 48 more
[2010-11-23 14:14:46,568] ERROR {org.apache.solr.core.SolrCore} -
REFCOUNT ERROR: unreferenced org.apache.solr.core.solrc...@58f24b6
(null) has a reference count of 1
[2010-11-23 14:14:46,568] ERROR {org.apache.solr.core.SolrCore} -
REFCOUNT ERROR: unreferenced org.apache.solr.core.solrc...@654dbbf6
(null) has a reference count of 1
[2010-11-23 14:14:46,568] ERROR {org.apache.solr.core.CoreContainer} -
CoreContainer was not shutdown prior to finalize(), indicates a bug --
POSSIBLE RESOURCE LEAK!!!
[2010-11-23 14:14:46,568] ERROR {org.apache.solr.core.CoreContainer} -
CoreContainer was not shutdown prior to finalize(), indicates a bug --
POSSIBLE RESOURCE LEAK!!!
--
Regards,
Tharindu
--
Regards,
Tharindu
--
Regards,
Tharindu
Re: Exceptions in Embedded Solr
Any help on this?
On Thu, Dec 2, 2010 at 7:51 PM, Tharindu Mathew mcclou...@gmail.com wrote:
Hi everyone,
I get the exception below when using Embedded Solr suddenly. If I
delete the Solr index it goes back to normal, but it obviously has to
start indexing from scratch. Any idea what the cause of this is?
java.lang.RuntimeException: java.io.FileNotFoundException:
/home/evanthika/WSO2/CARBON/GREG/3.6.0/23-11-2010/normal/wso2greg-3.6.0/solr/data/index/segments_2
(No such file or directory)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1068)
at org.apache.solr.core.SolrCore.init(SolrCore.java:579)
at
org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:137)
at
org.wso2.carbon.registry.indexing.solr.SolrClient.init(SolrClient.java:103)
at
org.wso2.carbon.registry.indexing.solr.SolrClient.getInstance(SolrClient.java:115)
... 44 more
Caused by: java.io.FileNotFoundException:
/home/evanthika/WSO2/CARBON/GREG/3.6.0/23-11-2010/normal/wso2greg-3.6.0/solr/data/index/segments_2
(No such file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.init(RandomAccessFile.java:212)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.init(SimpleFSDirectory.java:78)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.init(SimpleFSDirectory.java:108)
at
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.init(NIOFSDirectory.java:94)
at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:70)
at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:691)
at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:236)
at org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:72)
at
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:683)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:69)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:476)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:403)
at
org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:38)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1057)
... 48 more
[2010-11-23 14:14:46,568] ERROR {org.apache.solr.core.SolrCore} -
REFCOUNT ERROR: unreferenced org.apache.solr.core.solrc...@58f24b6
(null) has a reference count of 1
[2010-11-23 14:14:46,568] ERROR {org.apache.solr.core.SolrCore} -
REFCOUNT ERROR: unreferenced org.apache.solr.core.solrc...@654dbbf6
(null) has a reference count of 1
[2010-11-23 14:14:46,568] ERROR {org.apache.solr.core.CoreContainer} -
CoreContainer was not shutdown prior to finalize(), indicates a bug --
POSSIBLE RESOURCE LEAK!!!
[2010-11-23 14:14:46,568] ERROR {org.apache.solr.core.CoreContainer} -
CoreContainer was not shutdown prior to finalize(), indicates a bug --
POSSIBLE RESOURCE LEAK!!!
--
Regards,
Tharindu
--
Regards,
Tharindu
--
Regards,
Tharindu
Exceptions in Embedded Solr
Hi everyone,
I get the exception below when using Embedded Solr suddenly. If I
delete the Solr index it goes back to normal, but it obviously has to
start indexing from scratch. Any idea what the cause of this is?
java.lang.RuntimeException: java.io.FileNotFoundException:
/home/evanthika/WSO2/CARBON/GREG/3.6.0/23-11-2010/normal/wso2greg-3.6.0/solr/data/index/segments_2
(No such file or directory)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1068)
at org.apache.solr.core.SolrCore.init(SolrCore.java:579)
at
org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:137)
at org.wso2.carbon.registry.indexing.solr.SolrClient.init(SolrClient.java:103)
at
org.wso2.carbon.registry.indexing.solr.SolrClient.getInstance(SolrClient.java:115)
... 44 more
Caused by: java.io.FileNotFoundException:
/home/evanthika/WSO2/CARBON/GREG/3.6.0/23-11-2010/normal/wso2greg-3.6.0/solr/data/index/segments_2
(No such file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.init(RandomAccessFile.java:212)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.init(SimpleFSDirectory.java:78)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.init(SimpleFSDirectory.java:108)
at
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.init(NIOFSDirectory.java:94)
at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:70)
at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:691)
at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:236)
at org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:72)
at
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:683)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:69)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:476)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:403)
at
org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:38)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1057)
... 48 more
[2010-11-23 14:14:46,568] ERROR {org.apache.solr.core.SolrCore} -
REFCOUNT ERROR: unreferenced org.apache.solr.core.solrc...@58f24b6
(null) has a reference count of 1
[2010-11-23 14:14:46,568] ERROR {org.apache.solr.core.SolrCore} -
REFCOUNT ERROR: unreferenced org.apache.solr.core.solrc...@654dbbf6
(null) has a reference count of 1
[2010-11-23 14:14:46,568] ERROR {org.apache.solr.core.CoreContainer} -
CoreContainer was not shutdown prior to finalize(), indicates a bug --
POSSIBLE RESOURCE LEAK!!!
[2010-11-23 14:14:46,568] ERROR {org.apache.solr.core.CoreContainer} -
CoreContainer was not shutdown prior to finalize(), indicates a bug --
POSSIBLE RESOURCE LEAK!!!
--
Regards,
Tharindu
--
Regards,
Tharindu
Re: Jars required in classpath to run embedded solr server?
It requires all of the jars that are packed into solr.war. It is a full and complete implementation of indexing and searching. Tharindu Mathew wrote: Hi everyone, Do we need all lucene jars in the class path for this? Seems that the solr-solrj and solr-core jars are not enough (http://wiki.apache.org/solr/Solrj). It is asking for lucene jars in the classpath. Could I know what jars are required to run this? Thanks in advance.
Jars required in classpath to run embedded solr server?
Hi everyone, Do we need all lucene jars in the class path for this? Seems that the solr-solrj and solr-core jars are not enough (http://wiki.apache.org/solr/Solrj). It is asking for lucene jars in the classpath. Could I know what jars are required to run this? Thanks in advance. -- Regards, Tharindu
Re: Special Circumstances for embedded Solr
Any other commonly compelling reasons to use SolrJ? The most compelling reason (I think) is that if you program against the Solrj API, you can switch between embedded/http/streaming implementations without changing anything. This is great for our app that is either run as a small local instance of a big enterprise setting. ryan
Special Circumstances for embedded Solr
Hi all, We'd started using embedded Solr back in 2007, via a patched version of the in-progress 1.3 code base. I recently was reading http://wiki.apache.org/solr/EmbeddedSolr, and wondered about the paragraph that said: The simplest, safest, way to use Solr is via Solr's standard HTTP interfaces. Embedding Solr is less flexible, harder to support, not as well tested, and should be reserved for special circumstances. Given the current state of SolrJ, and the expected roadmap for Solr in general, what would be some guidelines for special circumstances that warrant the use of SolrJ? I know what ours were back in 2007 - namely: - we had multiple indexes, but didn't want to run multiple webapps (now handled by multi-core) - we needed efficient generation of updated indexes, without generating lots of HTTP traffic (now handled by DIH, maybe with specific extensions?) - we wanted tighter coupling of the front-end API with the back-end Solr search system, since this was an integrated system in the hands of customers - no just restart the webapp container option if anything got wedged (might still be an issue?) Any other commonly compelling reasons to use SolrJ? Thanks, -- Ken Ken Krugler +1 530-210-6378 http://bixolabs.com e l a s t i c w e b m i n i n g
Re: Embedded Solr search query
Thank you for the link Camen, It is a great help. Have a nice day Regards Eric On Fri, May 7, 2010 at 5:13 PM, caman aboxfortheotherst...@gmail.comwrote: I would just look at SOLR source code and see how standard search handler and dismaxSearchHandler are implemented. Look under package 'org.apache.solr. http://hudson.zones.apache.org/hudson/job/Solr-trunk/clover/org/apache/solr /handler/pkg-summary.htmlhttp://hudson.zones.apache.org/hudson/job/Solr-trunk/clover/org/apache/solr%0A/handler/pkg-summary.html handler' From: Eric Grobler [via Lucene] [mailto:ml-node+783212-2036924225-124...@n3.nabble.comml-node%2b783212-2036924225-124...@n3.nabble.com ] Sent: Friday, May 07, 2010 1:33 AM To: caman Subject: Re: Embedded Solr search query Hi Camen, I was hoping someone has done it already :-) I am also new to Solr/lucene, can you perhaps point me to a request handler example page? Thanks and Regards Eric On Fri, May 7, 2010 at 9:05 AM, caman [hidden email]wrote: Why not write a custom request handler which can parse, split, execute and combine results to your queries? From: Eric Grobler [via Lucene] [mailto:[hidden email][hidden email] ] Sent: Friday, May 07, 2010 1:01 AM To: caman Subject: Embedded Solr search query Hello Solr community, When a user search on our web page, we need to run 3 related but different queries. For SEO reasons, we cannot use Ajax so at the moment we run 3 queries sequentially inside a PHP script. Allthough Solr is superfast, the extra network overhead can make the 3 queries 400ms slower than it needs to be. Thus my question is: Is there a way whereby you can send 1 query string to Solr with 2 or more embedded search queries, where Solr will split and execute the queries and return the results of the multiple searches in 1 go. In other words, instead of: - send searchQuery1 get result1 - send searchQuery2 get result2 ... you run: - send searchQuery1+searchQuery2 - get result1+result2 Thanks and Regards Eric _ View message @ http://lucene.472066.n3.nabble.com/Embedded-Solr-search-query-tp783150p78315 0.html http://lucene.472066.n3.nabble.com/Embedded-Solr-search-query-tp78315 0p78315%0A0.htmlhttp://lucene.472066.n3.nabble.com/Embedded-Solr-search-query-tp78315%0A0p78315%0A0.html To start a new topic under Solr - User, email [hidden email][hidden email] To unsubscribe from Solr - User, click (link removed) GZvcnRoZW90aGVyc3R1ZmZAZ21haWwuY29tfDQ3MjA2OHwtOTM0OTI1NzEx here. -- View this message in context: http://lucene.472066.n3.nabble.com/Embedded-Solr-search-query-tp783150p78315 6.htmlhttp://lucene.472066.n3.nabble.com/Embedded-Solr-search-query-tp783150p78315%0A6.html Sent from the Solr - User mailing list archive at Nabble.com. _ View message @ http://lucene.472066.n3.nabble.com/Embedded-Solr-search-query-tp783150p78321 2.html To start a new topic under Solr - User, email ml-node+472068-464289649-124...@n3.nabble.comml-node%2b472068-464289649-124...@n3.nabble.com To unsubscribe from Solr - User, click (link removed) GZvcnRoZW90aGVyc3R1ZmZAZ21haWwuY29tfDQ3MjA2OHwtOTM0OTI1NzEx here. -- View this message in context: http://lucene.472066.n3.nabble.com/Embedded-Solr-search-query-tp783150p784098.html Sent from the Solr - User mailing list archive at Nabble.com.
Embedded Solr search query
Hello Solr community, When a user search on our web page, we need to run 3 related but different queries. For SEO reasons, we cannot use Ajax so at the moment we run 3 queries sequentially inside a PHP script. Allthough Solr is superfast, the extra network overhead can make the 3 queries 400ms slower than it needs to be. Thus my question is: Is there a way whereby you can send 1 query string to Solr with 2 or more embedded search queries, where Solr will split and execute the queries and return the results of the multiple searches in 1 go. In other words, instead of: - send searchQuery1 get result1 - send searchQuery2 get result2 ... you run: - send searchQuery1+searchQuery2 - get result1+result2 Thanks and Regards Eric
RE: Embedded Solr search query
Why not write a custom request handler which can parse, split, execute and combine results to your queries? From: Eric Grobler [via Lucene] [mailto:ml-node+783150-1027691461-124...@n3.nabble.com] Sent: Friday, May 07, 2010 1:01 AM To: caman Subject: Embedded Solr search query Hello Solr community, When a user search on our web page, we need to run 3 related but different queries. For SEO reasons, we cannot use Ajax so at the moment we run 3 queries sequentially inside a PHP script. Allthough Solr is superfast, the extra network overhead can make the 3 queries 400ms slower than it needs to be. Thus my question is: Is there a way whereby you can send 1 query string to Solr with 2 or more embedded search queries, where Solr will split and execute the queries and return the results of the multiple searches in 1 go. In other words, instead of: - send searchQuery1 get result1 - send searchQuery2 get result2 ... you run: - send searchQuery1+searchQuery2 - get result1+result2 Thanks and Regards Eric _ View message @ http://lucene.472066.n3.nabble.com/Embedded-Solr-search-query-tp783150p78315 0.html To start a new topic under Solr - User, email ml-node+472068-464289649-124...@n3.nabble.com To unsubscribe from Solr - User, click (link removed) GZvcnRoZW90aGVyc3R1ZmZAZ21haWwuY29tfDQ3MjA2OHwtOTM0OTI1NzEx here. -- View this message in context: http://lucene.472066.n3.nabble.com/Embedded-Solr-search-query-tp783150p783156.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Embedded Solr search query
Or send the queries in parallell from the PHP script (use CURL). Svein 2010/5/7 caman aboxfortheotherst...@gmail.com: Why not write a custom request handler which can parse, split, execute and combine results to your queries? From: Eric Grobler [via Lucene] [mailto:ml-node+783150-1027691461-124...@n3.nabble.com] Sent: Friday, May 07, 2010 1:01 AM To: caman Subject: Embedded Solr search query Hello Solr community, When a user search on our web page, we need to run 3 related but different queries. For SEO reasons, we cannot use Ajax so at the moment we run 3 queries sequentially inside a PHP script. Allthough Solr is superfast, the extra network overhead can make the 3 queries 400ms slower than it needs to be. Thus my question is: Is there a way whereby you can send 1 query string to Solr with 2 or more embedded search queries, where Solr will split and execute the queries and return the results of the multiple searches in 1 go. In other words, instead of: - send searchQuery1 get result1 - send searchQuery2 get result2 ... you run: - send searchQuery1+searchQuery2 - get result1+result2 Thanks and Regards Eric _ View message @ http://lucene.472066.n3.nabble.com/Embedded-Solr-search-query-tp783150p78315 0.html To start a new topic under Solr - User, email ml-node+472068-464289649-124...@n3.nabble.com To unsubscribe from Solr - User, click (link removed) GZvcnRoZW90aGVyc3R1ZmZAZ21haWwuY29tfDQ3MjA2OHwtOTM0OTI1NzEx here. -- View this message in context: http://lucene.472066.n3.nabble.com/Embedded-Solr-search-query-tp783150p783156.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Embedded Solr search query
Hi Camen, I was hoping someone has done it already :-) I am also new to Solr/lucene, can you perhaps point me to a request handler example page? Thanks and Regards Eric On Fri, May 7, 2010 at 9:05 AM, caman aboxfortheotherst...@gmail.comwrote: Why not write a custom request handler which can parse, split, execute and combine results to your queries? From: Eric Grobler [via Lucene] [mailto:ml-node+783150-1027691461-124...@n3.nabble.comml-node%2b783150-1027691461-124...@n3.nabble.com ] Sent: Friday, May 07, 2010 1:01 AM To: caman Subject: Embedded Solr search query Hello Solr community, When a user search on our web page, we need to run 3 related but different queries. For SEO reasons, we cannot use Ajax so at the moment we run 3 queries sequentially inside a PHP script. Allthough Solr is superfast, the extra network overhead can make the 3 queries 400ms slower than it needs to be. Thus my question is: Is there a way whereby you can send 1 query string to Solr with 2 or more embedded search queries, where Solr will split and execute the queries and return the results of the multiple searches in 1 go. In other words, instead of: - send searchQuery1 get result1 - send searchQuery2 get result2 ... you run: - send searchQuery1+searchQuery2 - get result1+result2 Thanks and Regards Eric _ View message @ http://lucene.472066.n3.nabble.com/Embedded-Solr-search-query-tp783150p78315 0.htmlhttp://lucene.472066.n3.nabble.com/Embedded-Solr-search-query-tp783150p78315%0A0.html To start a new topic under Solr - User, email ml-node+472068-464289649-124...@n3.nabble.comml-node%2b472068-464289649-124...@n3.nabble.com To unsubscribe from Solr - User, click (link removed) GZvcnRoZW90aGVyc3R1ZmZAZ21haWwuY29tfDQ3MjA2OHwtOTM0OTI1NzEx here. -- View this message in context: http://lucene.472066.n3.nabble.com/Embedded-Solr-search-query-tp783150p783156.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Embedded Solr search query
Hi Svein, Yes we thought of sending parallell queries, but you still have the extra network overhead. Regards Eric On Fri, May 7, 2010 at 9:11 AM, Svein Parnas sv...@trank.no wrote: Or send the queries in parallell from the PHP script (use CURL). Svein 2010/5/7 caman aboxfortheotherst...@gmail.com: Why not write a custom request handler which can parse, split, execute and combine results to your queries? From: Eric Grobler [via Lucene] [mailto:ml-node+783150-1027691461-124...@n3.nabble.comml-node%2b783150-1027691461-124...@n3.nabble.com ] Sent: Friday, May 07, 2010 1:01 AM To: caman Subject: Embedded Solr search query Hello Solr community, When a user search on our web page, we need to run 3 related but different queries. For SEO reasons, we cannot use Ajax so at the moment we run 3 queries sequentially inside a PHP script. Allthough Solr is superfast, the extra network overhead can make the 3 queries 400ms slower than it needs to be. Thus my question is: Is there a way whereby you can send 1 query string to Solr with 2 or more embedded search queries, where Solr will split and execute the queries and return the results of the multiple searches in 1 go. In other words, instead of: - send searchQuery1 get result1 - send searchQuery2 get result2 ... you run: - send searchQuery1+searchQuery2 - get result1+result2 Thanks and Regards Eric _ View message @ http://lucene.472066.n3.nabble.com/Embedded-Solr-search-query-tp783150p78315 0.html To start a new topic under Solr - User, email ml-node+472068-464289649-124...@n3.nabble.comml-node%2b472068-464289649-124...@n3.nabble.com To unsubscribe from Solr - User, click (link removed) GZvcnRoZW90aGVyc3R1ZmZAZ21haWwuY29tfDQ3MjA2OHwtOTM0OTI1NzEx here. -- View this message in context: http://lucene.472066.n3.nabble.com/Embedded-Solr-search-query-tp783150p783156.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: Embedded Solr search query
I would just look at SOLR source code and see how standard search handler and dismaxSearchHandler are implemented. Look under package 'org.apache.solr. http://hudson.zones.apache.org/hudson/job/Solr-trunk/clover/org/apache/solr /handler/pkg-summary.html handler' From: Eric Grobler [via Lucene] [mailto:ml-node+783212-2036924225-124...@n3.nabble.com] Sent: Friday, May 07, 2010 1:33 AM To: caman Subject: Re: Embedded Solr search query Hi Camen, I was hoping someone has done it already :-) I am also new to Solr/lucene, can you perhaps point me to a request handler example page? Thanks and Regards Eric On Fri, May 7, 2010 at 9:05 AM, caman [hidden email]wrote: Why not write a custom request handler which can parse, split, execute and combine results to your queries? From: Eric Grobler [via Lucene] [mailto:[hidden email][hidden email] ] Sent: Friday, May 07, 2010 1:01 AM To: caman Subject: Embedded Solr search query Hello Solr community, When a user search on our web page, we need to run 3 related but different queries. For SEO reasons, we cannot use Ajax so at the moment we run 3 queries sequentially inside a PHP script. Allthough Solr is superfast, the extra network overhead can make the 3 queries 400ms slower than it needs to be. Thus my question is: Is there a way whereby you can send 1 query string to Solr with 2 or more embedded search queries, where Solr will split and execute the queries and return the results of the multiple searches in 1 go. In other words, instead of: - send searchQuery1 get result1 - send searchQuery2 get result2 ... you run: - send searchQuery1+searchQuery2 - get result1+result2 Thanks and Regards Eric _ View message @ http://lucene.472066.n3.nabble.com/Embedded-Solr-search-query-tp783150p78315 0.htmlhttp://lucene.472066.n3.nabble.com/Embedded-Solr-search-query-tp78315 0p78315%0A0.html To start a new topic under Solr - User, email [hidden email][hidden email] To unsubscribe from Solr - User, click (link removed) GZvcnRoZW90aGVyc3R1ZmZAZ21haWwuY29tfDQ3MjA2OHwtOTM0OTI1NzEx here. -- View this message in context: http://lucene.472066.n3.nabble.com/Embedded-Solr-search-query-tp783150p78315 6.html Sent from the Solr - User mailing list archive at Nabble.com. _ View message @ http://lucene.472066.n3.nabble.com/Embedded-Solr-search-query-tp783150p78321 2.html To start a new topic under Solr - User, email ml-node+472068-464289649-124...@n3.nabble.com To unsubscribe from Solr - User, click (link removed) GZvcnRoZW90aGVyc3R1ZmZAZ21haWwuY29tfDQ3MjA2OHwtOTM0OTI1NzEx here. -- View this message in context: http://lucene.472066.n3.nabble.com/Embedded-Solr-search-query-tp783150p784098.html Sent from the Solr - User mailing list archive at Nabble.com.
Embedded solr - SLF4J exception
While attempting to work around my other issue, I'm trying to use an embedded solr server to try to programatically load data into solr. It seems though that I can't deploy my app, as a result of this exception: : java.lang.IllegalAccessError: tried to access field org.slf4j.impl.StaticLoggerBinder.SINGLETON from class org.slf4j.LoggerFactory I'm attempting to glassfish v3, which uses weld for CDI and I can only assume it's a conflict between SLF versions. Is it safe to upgrade to a newer SLF version? Thanks, John
Re: Embedded Solr problem
Hi Ranveer, I assume that you have enough knowlesge in Java. You should essentially your code for instantiating the server (depending on what you intend to do this may be done in a separate class or in a method of the class doing the queries). Then you use this instance to handle all the queries using for example the method query of SolrServer. For further information you may want to consult either the API documentation or the url http://wiki.apache.org/solr/Solrj from the wiki. Cheers, Sven --On Montag, 8. Februar 2010 08:53 +0530 Ranveer Kumar ranveer.s...@gmail.com wrote: Hi Sven, thanks for reply. yes i notice that every time when request, new instance is created of solr server. could you please guide me to do the same ( initialization to create an instance of SolrServer, once during first request). On Mon, Feb 8, 2010 at 2:11 AM, Sven Maurmann sven.maurm...@kippdata.dewrote: Hi, would it be possible that you instantiate a new instance of your SolrServer every time you do a query? You should use the code you quoted in your mail once during initialization to create an instance of SolrServer (the interface being implemented by EmbeddedSolrServer) and subsquently use the query method of SolrServer to do the query. Cheers, Sven --On Sonntag, 7. Februar 2010 21:54 +0530 Ranveer Kumar ranveer.s...@gmail.com wrote: Hi All, I am still very new to solr. Currently I am facing problem to use EmbeddedSolrServer. following is my code: File home = new File(D:/ranveer/java/solr_home/solr/first); CoreContainer coreContainer = new CoreContainer(); SolrConfig config = null; config = new SolrConfig(home + /core1,solrconfig.xml,null); CoreDescriptor descriptor = new CoreDescriptor(coreContainer,core1,home + /core1); SolrCore core = new SolrCore(core1, home+/core1/data, config, new IndexSchema(config, schema.xml,null), descriptor); coreContainer.register(core.getName(), core, true); final EmbeddedSolrServer server = new EmbeddedSolrServer(coreContainer, core1); Now my problem is every time when I making request for search SolrCore is initializing the core. I want if the core/instance of core is already start then just use previously started core. Due to this problem right now searching is taking too much time. I tried to close core after search but same thing when fresh search result is made, solr is starting from very basic. please help.. thanks
Embedded Solr problem
Hi All, I am still very new to solr. Currently I am facing problem to use EmbeddedSolrServer. following is my code: File home = new File(D:/ranveer/java/solr_home/solr/first); CoreContainer coreContainer = new CoreContainer(); SolrConfig config = null; config = new SolrConfig(home + /core1,solrconfig.xml,null); CoreDescriptor descriptor = new CoreDescriptor(coreContainer,core1,home + /core1); SolrCore core = new SolrCore(core1, home+/core1/data, config, new IndexSchema(config, schema.xml,null), descriptor); coreContainer.register(core.getName(), core, true); final EmbeddedSolrServer server = new EmbeddedSolrServer(coreContainer, core1); Now my problem is every time when I making request for search SolrCore is initializing the core. I want if the core/instance of core is already start then just use previously started core. Due to this problem right now searching is taking too much time. I tried to close core after search but same thing when fresh search result is made, solr is starting from very basic. please help.. thanks
Re: Embedded Solr problem
Hi, would it be possible that you instantiate a new instance of your SolrServer every time you do a query? You should use the code you quoted in your mail once during initialization to create an instance of SolrServer (the interface being implemented by EmbeddedSolrServer) and subsquently use the query method of SolrServer to do the query. Cheers, Sven --On Sonntag, 7. Februar 2010 21:54 +0530 Ranveer Kumar ranveer.s...@gmail.com wrote: Hi All, I am still very new to solr. Currently I am facing problem to use EmbeddedSolrServer. following is my code: File home = new File(D:/ranveer/java/solr_home/solr/first); CoreContainer coreContainer = new CoreContainer(); SolrConfig config = null; config = new SolrConfig(home + /core1,solrconfig.xml,null); CoreDescriptor descriptor = new CoreDescriptor(coreContainer,core1,home + /core1); SolrCore core = new SolrCore(core1, home+/core1/data, config, new IndexSchema(config, schema.xml,null), descriptor); coreContainer.register(core.getName(), core, true); final EmbeddedSolrServer server = new EmbeddedSolrServer(coreContainer, core1); Now my problem is every time when I making request for search SolrCore is initializing the core. I want if the core/instance of core is already start then just use previously started core. Due to this problem right now searching is taking too much time. I tried to close core after search but same thing when fresh search result is made, solr is starting from very basic. please help.. thanks
Re: Embedded Solr problem
Hi Sven, thanks for reply. yes i notice that every time when request, new instance is created of solr server. could you please guide me to do the same ( initialization to create an instance of SolrServer, once during first request). On Mon, Feb 8, 2010 at 2:11 AM, Sven Maurmann sven.maurm...@kippdata.dewrote: Hi, would it be possible that you instantiate a new instance of your SolrServer every time you do a query? You should use the code you quoted in your mail once during initialization to create an instance of SolrServer (the interface being implemented by EmbeddedSolrServer) and subsquently use the query method of SolrServer to do the query. Cheers, Sven --On Sonntag, 7. Februar 2010 21:54 +0530 Ranveer Kumar ranveer.s...@gmail.com wrote: Hi All, I am still very new to solr. Currently I am facing problem to use EmbeddedSolrServer. following is my code: File home = new File(D:/ranveer/java/solr_home/solr/first); CoreContainer coreContainer = new CoreContainer(); SolrConfig config = null; config = new SolrConfig(home + /core1,solrconfig.xml,null); CoreDescriptor descriptor = new CoreDescriptor(coreContainer,core1,home + /core1); SolrCore core = new SolrCore(core1, home+/core1/data, config, new IndexSchema(config, schema.xml,null), descriptor); coreContainer.register(core.getName(), core, true); final EmbeddedSolrServer server = new EmbeddedSolrServer(coreContainer, core1); Now my problem is every time when I making request for search SolrCore is initializing the core. I want if the core/instance of core is already start then just use previously started core. Due to this problem right now searching is taking too much time. I tried to close core after search but same thing when fresh search result is made, solr is starting from very basic. please help.. thanks
Re: Embedded solr with third party libraries
: distirbution. When we run test cases our schema.xml has defintion for lucid : kstem and it throws ClassNotFound Exception. : We declared the depency for the two jars lucid-kstem.jar and : lucid-solr-kstem.jar but still it throws an error. explain what you mean by declared the depency ? : C:\DOCUME~1\username\LOCALS~1\Temp\solr-all\0.8194571792905493\solr\conf\schema.xml : : Now in order for the jar to be loaded should i copy the two jars to solr/lib : directory. is that the default location embedded solr looks into for some : default jars. assuming C:\DOCUME~1\username\LOCALS~1\Temp\solr-all\0.8194571792905493\solr is your sole home dir, then yes you can copy your jars into C:\DOCUME~1\username\LOCALS~1\Temp\solr-all\0.8194571792905493\solr\lib and that should work ... or starting in Solr 1.4 you can use the lib directorives to specify a jar anywhere on disk. see the example solrconfig.xml for the syntax. -Hoss
logger in embedded solr
Hi all, I have an J2EE application using embedded solr via solr4j. It seems the logging that SOLR produces has a mind of its own, and is not changeable via my log4j.properties. In fact I know this because I wired in a Log4J config listener in my web.xml and redirected all my logs to a custom location. Which works, but now all my messages go to the custom location and all the embedded SOLR messages are still going into catalina.out. How can I get access to the logger of the Embedded SOLR. Thanks, Tim Harsch Sr. Software Engineer Perot Systems
Re: logger in embedded solr
check: http://wiki.apache.org/solr/SolrLogging if you are using 1.4 you want to drop in the slf4j-log4j jar file and then it should read your log4j configs On Nov 19, 2009, at 2:15 PM, Harsch, Timothy J. (ARC-TI)[PEROT SYSTEMS] wrote: Hi all, I have an J2EE application using embedded solr via solr4j. It seems the logging that SOLR produces has a mind of its own, and is not changeable via my log4j.properties. In fact I know this because I wired in a Log4J config listener in my web.xml and redirected all my logs to a custom location. Which works, but now all my messages go to the custom location and all the embedded SOLR messages are still going into catalina.out. How can I get access to the logger of the Embedded SOLR. Thanks, Tim Harsch Sr. Software Engineer Perot Systems
Embedded Solr, creating new cores programatically
Hi folks, I am working on a project in which we would like to create new cores programatically within an embedded solr instance. My question is: How much of the directory/configuration file gruntwork do I have to do myself? I figure I have to create the solr.home directory myself, but do I have to create solr.xml, as well as directories for each core I would like to run? Do I have to create the configuration directory for each core manually, as well as populate the required configuration files manually? Thank you all, Jay Shollenberger
Re: Embedded Solr Clustering
: Yes we are using Solr for a non-traditional search purpose and the : performance is critical. However it sounds like that sharing the same index : could slow down reading / writing to the index. And access synchronization : is tricky as well. no matter how you use Solr (HTTP or Embedded) only one SOlrCore can be writting to an index at a time. : Therefore, we might have to use a single web based Solr instance or use : multiple embedded Solr instances and setup the script based replication. not clear if this is what you mena: but you can always have a single solr master (which might be embedded or it might be the war) and then use script replication or shared disk to make that index available for numerous additional solr instance (again: either embedded or war based) for queries. -Hoss
Re: Embedded Solr Clustering
On Mon, Aug 10, 2009 at 3:57 AM, born2fish tswan...@yahoo.com wrote: Hi everyone, We have a web app that uses embedded solr for better performance. I would advise against it. We use Solr on sites with millions of page views a month on HTTP. With HTTP keep-alives, the overhead of an http request is minimal as compared to the actual search. You have the advantages of replication as well as the option of adding a http cache in front of Solr. Is there a performance problem you're trying to solve by using embedded solr? Now we are trying to deploy the app to a clustered environment. My question is: 1. Can we configure the embedded solr instances to share the same index on the network? Yes. It may be very slow though. Best to benchmark it before going to production. Also, you'll need to make sure that only one Solr instance is writing to the index at one time. It is better to have separate indexes. 2. If the answer to question 1 is no, can we configure embedded solr instances to replicate indexes in a master / slave fashion just like normal web based Solr? Yes you can use the script based replication. You'd need to expose a way to call commit on your application if you use embedded solr. -- Regards, Shalin Shekhar Mangar.
Re: Embedded Solr Clustering
Thanks Shalin and Avlesh for your responses. Yes we are using Solr for a non-traditional search purpose and the performance is critical. However it sounds like that sharing the same index could slow down reading / writing to the index. And access synchronization is tricky as well. Therefore, we might have to use a single web based Solr instance or use multiple embedded Solr instances and setup the script based replication. Thanks again for your help! born2fish wrote: Hi everyone, We have a web app that uses embedded solr for better performance. Now we are trying to deploy the app to a clustered environment. My question is: 1. Can we configure the embedded solr instances to share the same index on the network? 2. If the answer to question 1 is no, can we configure embedded solr instances to replicate indexes in a master / slave fashion just like normal web based Solr? Thanks, born2fish -- View this message in context: http://www.nabble.com/Embedded-Solr-Clustering-tp24891931p24900854.html Sent from the Solr - User mailing list archive at Nabble.com.
Embedded Solr Clustering
Hi everyone, We have a web app that uses embedded solr for better performance. Now we are trying to deploy the app to a clustered environment. My question is: 1. Can we configure the embedded solr instances to share the same index on the network? 2. If the answer to question 1 is no, can we configure embedded solr instances to replicate indexes in a master / slave fashion just like normal web based Solr? Thanks, born2fish -- View this message in context: http://www.nabble.com/Embedded-Solr-Clustering-tp24891931p24891931.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Embedded Solr Clustering
Can we configure the embedded solr instances to share the same index on the network? Yes, multiple EmbeddedSolrServer instances can point to the same core. Cheers Avlesh On Mon, Aug 10, 2009 at 3:57 AM, born2fish tswan...@yahoo.com wrote: Hi everyone, We have a web app that uses embedded solr for better performance. Now we are trying to deploy the app to a clustered environment. My question is: 1. Can we configure the embedded solr instances to share the same index on the network? 2. If the answer to question 1 is no, can we configure embedded solr instances to replicate indexes in a master / slave fashion just like normal web based Solr? Thanks, born2fish -- View this message in context: http://www.nabble.com/Embedded-Solr-Clustering-tp24891931p24891931.html Sent from the Solr - User mailing list archive at Nabble.com.
Transfer of Index Vs HTTP GET Vs Embedded Solr -- Urgent Help
Hi, I have a search engine on Solr. Also I have a remote web application which will be using the Solr Indexes for search. I have three scenarios: 1) Transfer the Indexes to the Remote Application. - This will reduce load on the actual solr server and make seraches faster. - Need to write some code to transfer the index - Need to double my effort to update,merge,optimize index 2)Use HTTP GET - Will increase load on the Solr server - No extra code needed for transfer 3) Embedded Serach - Use SolrJ for querying I want to know which is the best approach. Regards, Ninad Raut.
Re: Transfer of Index Vs HTTP GET Vs Embedded Solr -- Urgent Help
On Thu, Aug 6, 2009 at 12:24 PM, Ninad Rauthbase.user.ni...@gmail.com wrote: Hi, I have a search engine on Solr. Also I have a remote web application which will be using the Solr Indexes for search. I have three scenarios: 1) Transfer the Indexes to the Remote Application. - This will reduce load on the actual solr server and make seraches faster. - Need to write some code to transfer the index - Need to double my effort to update,merge,optimize index 2)Use HTTP GET - Will increase load on the Solr server - No extra code needed for transfer This by far is the best solution. ecause you do not have to do any work at all. It all works out of the box and that is what everyone uses 3) Embedded Serach - Use SolrJ for querying I want to know which is the best approach. Regards, Ninad Raut. -- - Noble Paul | Principal Engineer| AOL | http://aol.com
Re: Transfer of Index Vs HTTP GET Vs Embedded Solr -- Urgent Help
Hi Noble, Can you explain a bit more on how to use Solr out of the box. I am looking at ways to design the UI for remote application quickly and with less problems. Also could you elaborate more on what can go wrong with the first option? Thanks. 2009/8/6 Noble Paul നോബിള് नोब्ळ् noble.p...@corp.aol.com On Thu, Aug 6, 2009 at 12:24 PM, Ninad Rauthbase.user.ni...@gmail.com wrote: Hi, I have a search engine on Solr. Also I have a remote web application which will be using the Solr Indexes for search. I have three scenarios: 1) Transfer the Indexes to the Remote Application. - This will reduce load on the actual solr server and make seraches faster. - Need to write some code to transfer the index - Need to double my effort to update,merge,optimize index 2)Use HTTP GET - Will increase load on the Solr server - No extra code needed for transfer This by far is the best solution. ecause you do not have to do any work at all. It all works out of the box and that is what everyone uses 3) Embedded Serach - Use SolrJ for querying I want to know which is the best approach. Regards, Ninad Raut. -- - Noble Paul | Principal Engineer| AOL | http://aol.com
Re: Transfer of Index Vs HTTP GET Vs Embedded Solr -- Urgent Help
About the first option, caches are more effective with more traffic, so ten front end servers using three Solr servers will have better caching and probably better overall performance than having separate search on all ten servers. You can even put an HTTP cache in there and get better caching. Cached HTTP responses are usually faster than accessing disc locally. You say you have a remote web application. How remote? If the indexes are big, then copying them to a remote location is a lot of traffic. wunder On Aug 6, 2009, at 8:49 PM, Ninad Raut wrote: Hi Noble, Can you explain a bit more on how to use Solr out of the box. I am looking at ways to design the UI for remote application quickly and with less problems. Also could you elaborate more on what can go wrong with the first option? Thanks. 2009/8/6 Noble Paul നോബിള് नोब्ळ् noble.p...@corp.aol.com On Thu, Aug 6, 2009 at 12:24 PM, Ninad Rauthbase.user.ni...@gmail.com wrote: Hi, I have a search engine on Solr. Also I have a remote web application which will be using the Solr Indexes for search. I have three scenarios: 1) Transfer the Indexes to the Remote Application. - This will reduce load on the actual solr server and make seraches faster. - Need to write some code to transfer the index - Need to double my effort to update,merge,optimize index 2)Use HTTP GET - Will increase load on the Solr server - No extra code needed for transfer This by far is the best solution. ecause you do not have to do any work at all. It all works out of the box and that is what everyone uses 3) Embedded Serach - Use SolrJ for querying I want to know which is the best approach. Regards, Ninad Raut. -- - Noble Paul | Principal Engineer| AOL | http://aol.com
Re: Transfer of Index Vs HTTP GET Vs Embedded Solr -- Urgent Help
The remote web app will be accessing the Solr server via internet. Its not a intranet setup. On Fri, Aug 7, 2009 at 10:19 AM, Walter Underwood wun...@wunderwood.orgwrote: About the first option, caches are more effective with more traffic, so ten front end servers using three Solr servers will have better caching and probably better overall performance than having separate search on all ten servers. You can even put an HTTP cache in there and get better caching. Cached HTTP responses are usually faster than accessing disc locally. You say you have a remote web application. How remote? If the indexes are big, then copying them to a remote location is a lot of traffic. wunder On Aug 6, 2009, at 8:49 PM, Ninad Raut wrote: Hi Noble, Can you explain a bit more on how to use Solr out of the box. I am looking at ways to design the UI for remote application quickly and with less problems. Also could you elaborate more on what can go wrong with the first option? Thanks. 2009/8/6 Noble Paul നോബിള് नोब्ळ् noble.p...@corp.aol.com On Thu, Aug 6, 2009 at 12:24 PM, Ninad Rauthbase.user.ni...@gmail.com wrote: Hi, I have a search engine on Solr. Also I have a remote web application which will be using the Solr Indexes for search. I have three scenarios: 1) Transfer the Indexes to the Remote Application. - This will reduce load on the actual solr server and make seraches faster. - Need to write some code to transfer the index - Need to double my effort to update,merge,optimize index 2)Use HTTP GET - Will increase load on the Solr server - No extra code needed for transfer This by far is the best solution. ecause you do not have to do any work at all. It all works out of the box and that is what everyone uses 3) Embedded Serach - Use SolrJ for querying I want to know which is the best approach. Regards, Ninad Raut. -- - Noble Paul | Principal Engineer| AOL | http://aol.com
Re: Transfer of Index Vs HTTP GET Vs Embedded Solr -- Urgent Help
The you should consider replicating the index to the local intranet and still run that it as a separate app. On Fri, Aug 7, 2009 at 10:53 AM, Ninad Rauthbase.user.ni...@gmail.com wrote: The remote web app will be accessing the Solr server via internet. Its not a intranet setup. On Fri, Aug 7, 2009 at 10:19 AM, Walter Underwood wun...@wunderwood.orgwrote: About the first option, caches are more effective with more traffic, so ten front end servers using three Solr servers will have better caching and probably better overall performance than having separate search on all ten servers. You can even put an HTTP cache in there and get better caching. Cached HTTP responses are usually faster than accessing disc locally. You say you have a remote web application. How remote? If the indexes are big, then copying them to a remote location is a lot of traffic. wunder On Aug 6, 2009, at 8:49 PM, Ninad Raut wrote: Hi Noble, Can you explain a bit more on how to use Solr out of the box. I am looking at ways to design the UI for remote application quickly and with less problems. Also could you elaborate more on what can go wrong with the first option? Thanks. 2009/8/6 Noble Paul നോബിള് नोब्ळ् noble.p...@corp.aol.com On Thu, Aug 6, 2009 at 12:24 PM, Ninad Rauthbase.user.ni...@gmail.com wrote: Hi, I have a search engine on Solr. Also I have a remote web application which will be using the Solr Indexes for search. I have three scenarios: 1) Transfer the Indexes to the Remote Application. - This will reduce load on the actual solr server and make seraches faster. - Need to write some code to transfer the index - Need to double my effort to update,merge,optimize index 2)Use HTTP GET - Will increase load on the Solr server - No extra code needed for transfer This by far is the best solution. ecause you do not have to do any work at all. It all works out of the box and that is what everyone uses 3) Embedded Serach - Use SolrJ for querying I want to know which is the best approach. Regards, Ninad Raut. -- - Noble Paul | Principal Engineer| AOL | http://aol.com -- - Noble Paul | Principal Engineer| AOL | http://aol.com
Re: Transfer of Index Vs HTTP GET Vs Embedded Solr -- Urgent Help
The you should consider replicating the index to the local intranet and still run that it as a separate app. Will it be the same master-slave replication?? If the master is multicore, can I specifically replicate an index of a certain core ? Thanks for the help. 2009/8/7 Noble Paul നോബിള് नोब्ळ् noble.p...@corp.aol.com The you should consider replicating the index to the local intranet and still run that it as a separate app. On Fri, Aug 7, 2009 at 10:53 AM, Ninad Rauthbase.user.ni...@gmail.com wrote: The remote web app will be accessing the Solr server via internet. Its not a intranet setup. On Fri, Aug 7, 2009 at 10:19 AM, Walter Underwood wun...@wunderwood.org wrote: About the first option, caches are more effective with more traffic, so ten front end servers using three Solr servers will have better caching and probably better overall performance than having separate search on all ten servers. You can even put an HTTP cache in there and get better caching. Cached HTTP responses are usually faster than accessing disc locally. You say you have a remote web application. How remote? If the indexes are big, then copying them to a remote location is a lot of traffic. wunder On Aug 6, 2009, at 8:49 PM, Ninad Raut wrote: Hi Noble, Can you explain a bit more on how to use Solr out of the box. I am looking at ways to design the UI for remote application quickly and with less problems. Also could you elaborate more on what can go wrong with the first option? Thanks. 2009/8/6 Noble Paul നോബിള് नोब्ळ् noble.p...@corp.aol.com On Thu, Aug 6, 2009 at 12:24 PM, Ninad Raut hbase.user.ni...@gmail.com wrote: Hi, I have a search engine on Solr. Also I have a remote web application which will be using the Solr Indexes for search. I have three scenarios: 1) Transfer the Indexes to the Remote Application. - This will reduce load on the actual solr server and make seraches faster. - Need to write some code to transfer the index - Need to double my effort to update,merge,optimize index 2)Use HTTP GET - Will increase load on the Solr server - No extra code needed for transfer This by far is the best solution. ecause you do not have to do any work at all. It all works out of the box and that is what everyone uses 3) Embedded Serach - Use SolrJ for querying I want to know which is the best approach. Regards, Ninad Raut. -- - Noble Paul | Principal Engineer| AOL | http://aol.com -- - Noble Paul | Principal Engineer| AOL | http://aol.com
Embedded Solr weird behaviour
Hello, I am using both Solr server and Solr embedded versions in the same context. I am using the Solr Server for indexing data which can be accessed at enterprise level, and the embedded version in a desktop application. The idea is that both index the same data, have the same schema.xml and config. My problem: when querying both versions I get different results for this case: query=adventure AND category:Publishing Industry Please note that 'Publishing Industry is actually composed by 2 words. For the Server version it works very well, for the Embedded version, I get no result. In this case: query=adventure AND category:Book - I get correct results with both version. category is a field type in my schema. I noticed that when I have something like: AND category:'composed words', the Embedded version fails. In the schema I tried making the category fieldType as text, string, etc, but no results any suggestion would be very appreciated. Thanks, Adrian
Re: Embedded Solr updates not showing until restart
Grant Ingersoll-6 wrote: Can you share your code? Or reduce it down to a repeatable test? I'll try to do this. For now I'm proceeding with the HTTP route. We're going to want to revisit this and I'll likely do it at that time. Thanks, Erik -- View this message in context: http://www.nabble.com/Embedded-Solr-updates-not-showing-until-restart-tp21546235p21614923.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Embedded Solr updates not showing until restart
Grant Ingersoll-6 wrote: Do they show up if you use non-embedded? That is, if you hit that slave over HTTP from your browser, are the changes showing up? Yes. Changing the config to access the server over HTTP works fine. When looking at our console logs for the Solr Server, I can see no discernable difference between the embedded and HTTP approaches. The snapinstaller appears to be working in both cases, but changes to the index don't show up in queries when the slave is configured as embedded. I'm moving forward with the HTTP approach, but the embedded approach is desirable for two (obvious) reasons: 1) performance improvement, 2) simpler deployment. Thanks. -- View this message in context: http://www.nabble.com/Embedded-Solr-updates-not-showing-until-restart-tp21546235p21562955.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Embedded Solr updates not showing until restart
Can you share your code? Or reduce it down to a repeatable test? On Jan 20, 2009, at 8:22 AM, edre...@ha wrote: Grant Ingersoll-6 wrote: Do they show up if you use non-embedded? That is, if you hit that slave over HTTP from your browser, are the changes showing up? Yes. Changing the config to access the server over HTTP works fine. When looking at our console logs for the Solr Server, I can see no discernable difference between the embedded and HTTP approaches. The snapinstaller appears to be working in both cases, but changes to the index don't show up in queries when the slave is configured as embedded. I'm moving forward with the HTTP approach, but the embedded approach is desirable for two (obvious) reasons: 1) performance improvement, 2) simpler deployment. Thanks. -- View this message in context: http://www.nabble.com/Embedded-Solr-updates-not-showing-until-restart-tp21546235p21562955.html Sent from the Solr - User mailing list archive at Nabble.com. -- Grant Ingersoll Lucene Helpful Hints: http://wiki.apache.org/lucene-java/BasicsOfPerformance http://wiki.apache.org/lucene-java/LuceneFAQ
Embedded Solr updates not showing until restart
Hi, We're evaluating the use of Solr for use in a web application. I've got the web application configured to use an embedded instance of Solr for queries (setup as a slave), and a remote instance for writes (setup as a master). The replication scripts are running fine and the embedded slave does appear to be getting the updates, but queries run against the embedded slave don't show up until I restart the web application. We're using SolrJ as our interface to Solr. Can anyone provide any insight into why updates don't show up until after a webapp restart? Thanks, Erik -- View this message in context: http://www.nabble.com/Embedded-Solr-updates-not-showing-until-restart-tp21546235p21546235.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Embedded Solr updates not showing until restart
Do they show up if you use non-embedded? That is, if you hit that slave over HTTP from your browser, are the changes showing up? On Jan 19, 2009, at 11:18 AM, edre...@ha wrote: Hi, We're evaluating the use of Solr for use in a web application. I've got the web application configured to use an embedded instance of Solr for queries (setup as a slave), and a remote instance for writes (setup as a master). The replication scripts are running fine and the embedded slave does appear to be getting the updates, but queries run against the embedded slave don't show up until I restart the web application. We're using SolrJ as our interface to Solr. Can anyone provide any insight into why updates don't show up until after a webapp restart? Thanks, Erik -- View this message in context: http://www.nabble.com/Embedded-Solr-updates-not-showing-until-restart-tp21546235p21546235.html Sent from the Solr - User mailing list archive at Nabble.com. -- Grant Ingersoll Lucene Helpful Hints: http://wiki.apache.org/lucene-java/BasicsOfPerformance http://wiki.apache.org/lucene-java/LuceneFAQ
Re: Using embedded Solr with admin GUI
Ken Krugler wrote: If you are running a single webapp, you can just put the jsp files in there. I'm guessing that isn't what you mean though. Well, ultimately we're heading towards a single webapp with multiple embedded Solr cores. In that case, could the .jsp-based GUI/admin functionality peacefully co-exist with our use of the embedded cores? yes, this is how the multi-core support in trunk works. There are a bunch of admin request handlers that do many of the things from the /admin/ jsp files without the nice interface. The one major missing component was analysis.jsp, but grant just added: https://issues.apache.org/jira/browse/SOLR-477 Is there a description of the roadmap for the Solr GUI? For example, I'm assuming the .jsp files will still exist going forward, but will become much more of just a GUI layer on top of the new/beefed up admin request handlers - yes? Or is the plan to eventually get to just Javascript on HTML using JSON responses from these request handlers? No formal description -- my hope is to remove as much scriptlet code as possible and make the jsp a simple GUI layer. I briefly thought changing most admin pages to handler+xslt was a good idea. But the flexibility to use jsp is nice. ryan Thanks, -- Ken Ken Krugler wrote: Hi all, We're moving towards embedding multiple Solr cores, versus using multiple Solr webapps, as a way of simplifying our build/deploy and also getting more control over the startup/update process. But I'd hate to lose that handy GUI for inspecting the schema and (most importantly) trying out queries with explain turned on. Has anybody tried this dual-mode method of operation? Thoughts on whether it's workable, and what the issues would be? I've taken a quick look at the .jsp and supporting Java code, and have some ideas on what would be needed, but I'm hoping there's an easy(er) approach than just whacking at the admin support code. Thanks, -- Ken
Re: Using embedded Solr with admin GUI
If you are running a single webapp, you can just put the jsp files in there. I'm guessing that isn't what you mean though. Well, ultimately we're heading towards a single webapp with multiple embedded Solr cores. In that case, could the .jsp-based GUI/admin functionality peacefully co-exist with our use of the embedded cores? There are a bunch of admin request handlers that do many of the things from the /admin/ jsp files without the nice interface. The one major missing component was analysis.jsp, but grant just added: https://issues.apache.org/jira/browse/SOLR-477 Is there a description of the roadmap for the Solr GUI? For example, I'm assuming the .jsp files will still exist going forward, but will become much more of just a GUI layer on top of the new/beefed up admin request handlers - yes? Or is the plan to eventually get to just Javascript on HTML using JSON responses from these request handlers? Thanks, -- Ken Ken Krugler wrote: Hi all, We're moving towards embedding multiple Solr cores, versus using multiple Solr webapps, as a way of simplifying our build/deploy and also getting more control over the startup/update process. But I'd hate to lose that handy GUI for inspecting the schema and (most importantly) trying out queries with explain turned on. Has anybody tried this dual-mode method of operation? Thoughts on whether it's workable, and what the issues would be? I've taken a quick look at the .jsp and supporting Java code, and have some ideas on what would be needed, but I'm hoping there's an easy(er) approach than just whacking at the admin support code. Thanks, -- Ken -- Ken Krugler Krugle, Inc. +1 530-210-6378 If you can't find it, you can't fix it
Re: Using embedded Solr with admin GUI
If you are running a single webapp, you can just put the jsp files in there. I'm guessing that isn't what you mean though. There are a bunch of admin request handlers that do many of the things from the /admin/ jsp files without the nice interface. The one major missing component was analysis.jsp, but grant just added: https://issues.apache.org/jira/browse/SOLR-477 Ken Krugler wrote: Hi all, We're moving towards embedding multiple Solr cores, versus using multiple Solr webapps, as a way of simplifying our build/deploy and also getting more control over the startup/update process. But I'd hate to lose that handy GUI for inspecting the schema and (most importantly) trying out queries with explain turned on. Has anybody tried this dual-mode method of operation? Thoughts on whether it's workable, and what the issues would be? I've taken a quick look at the .jsp and supporting Java code, and have some ideas on what would be needed, but I'm hoping there's an easy(er) approach than just whacking at the admin support code. Thanks, -- Ken
Using embedded Solr with admin GUI
Hi all, We're moving towards embedding multiple Solr cores, versus using multiple Solr webapps, as a way of simplifying our build/deploy and also getting more control over the startup/update process. But I'd hate to lose that handy GUI for inspecting the schema and (most importantly) trying out queries with explain turned on. Has anybody tried this dual-mode method of operation? Thoughts on whether it's workable, and what the issues would be? I've taken a quick look at the .jsp and supporting Java code, and have some ideas on what would be needed, but I'm hoping there's an easy(er) approach than just whacking at the admin support code. Thanks, -- Ken -- Ken Krugler Krugle, Inc. +1 530-210-6378 If you can't find it, you can't fix it
Embedded Solr and Servlet Solr
Hi! I am using Solr (the classic servlet one) in my application. But for special cases, and just for the sake of optimization, I would like to access it locally so I can avoid HTTP calls and XML serializing. This is for full re indexation of Solr, which is much faster in embedded environment than servlet. Is it possible to use SolrJ in my web application to access Solr remotely and use SolrJ in a simple application to access Solr locally for full re indexation? Thanks! Jonathan
