Re: using maven to deploy solr on tomcat
okay this looks promising. I will give it a try and let you know how it
goes. Thanks
On Wed, Apr 17, 2013 at 9:19 AM, jnduan wrote:
> hi Adeel,
> I have use solr with maven since 2011,and my dependency is not solr but
> solr-core and some other dependencies .
> therefore,my project structure is just like unpack the solr.war file with
> out the dir 'WEB-INF/lib'.
>
> So I can write some code work with solr ,e.g. a listener set up system
> properties 'solr.solr.home' to the right path before solr is lunch.Even set
> up some properties for zookeeper in solr 4.2.1.
>
> You said your solr home's source path is
> 'src/main/resources/solr-dev/',that means the solr home is
> 'WEB-INF/classes/solr-dev' at the runtime.Then just use
> ClassLoader.getResource() in a ServletContextListener to get the absolute
> path of solr home ,just like this:
>
> private static final String SOLR_HOME_PROP_NAME = "solr.solr.home";
>
> .
>
> try {
> URL url =
> this.getClass().getClassLoader().getResource("/solr-dev");
> if (url != null) {
> File file = new File(url.toURI());
> if (file.exists()) {
>
> System.setProperty(SOLR_HOME_PROP_NAME,file.getCanonicalPath());
> logger.info(" Set /solr/home to system
> properties,key:{}, value:{}",SOLR_HOME_PROP_NAME,file.getCanonicalPath());
> }
> else{
> logger.error("Resouce url '/solr' is not exists");
> }
> }
> else{
> logger.error("Can not locate resource url '/solr'");
> }
> } catch (Exception e) {
> logger.error(e.getMessage(), e);
> }
>
> I wish this could help,best regards.
>
> Duan Jienan
> 在 2013-4-16，上午4:33，Adeel Qureshi 写道：
>
> > I am trying to embed solr war in an empty application to be able to use
> > maven to deploy the generated war file (that includes solr war) to
> tomcat.
> > My pom file is simple
> >
> >
> >
> > org.apache.solr
> > solr
> > ${solr.version}
> > war
> >
> >
> >
> > this brings in the solr war but now i need to specify solr home to choose
> > between a dev and prod directory as the solr home. So I have setup a env
> > variable
> >
> > > value="src/main/resources/solr-dev"/>
> >
> > but this leads to absolute path of
> >
> > INFO: Using JNDI solr.home: src/main/resources/solr-dev
> > INFO: looking for solr.xml:
> > C:\springsource\sts-2.8.1.RELEASE\src\main\resources\solr-dev\solr.xml
> >
> > notice that its uses the given path as absolute path and obviously doesnt
> > ends up looking inside the application directory for these config
> directory.
> >
> > Any help would be appreciate.
> > Thanks
> > Adeel
>
>
Re: using maven to deploy solr on tomcat
the problem is i need to deploy it on servers where i dont know what the absolute path will be .. basically my goal is to load solr with a different set of configuration files based on the environment its in. Is there a a better different way to do this On Mon, Apr 15, 2013 at 11:29 PM, Shawn Heisey wrote: > On 4/15/2013 2:33 PM, Adeel Qureshi wrote: > > > value="src/main/resources/solr-dev"/> > > > > but this leads to absolute path of > > > > INFO: Using JNDI solr.home: src/main/resources/solr-dev > > INFO: looking for solr.xml: > > C:\springsource\sts-2.8.1.RELEASE\src\main\resources\solr-dev\solr.xml > > If you use a relative path for the solr home as you have done, it will > be relative to the current working directory. The CWD can vary > depending on how tomcat gets started. In your case, the CWD seems to be > "C:\springsource\sts-2.8.1.RELEASE". If you change the CWD in the > tomcat startup, you will probably have to set the TOMCAT_HOME > environment variable for tomcat to start correctly, so I don't recommend > doing that. > > It is usually best to choose an absolute path for the solr home. Solr > will find solr.xml there, which it will use to find the rest of your > config(s). All paths in solr.xml and other solr config files can be > relative. > > What you are seeing as an absolute path is likely the current working > directory plus your solr home setting. > > Thanks, > Shawn > >
Re: solr cloud concepts
Thanks for the explanation. It makes sense but I am hoping that you can clarify things a bit more .. so now it sounds like in solrcloud the concept of cores have changed a bit .. as you explained that for me to have 2 cores with different schemas I will need 2 different collections .. and one good thing about solrcores was that you could create new ones with coreadmin api or the http calls .. to create new collections its not that automated right .. secondly if collections represent what kind of used to be solrcore then once i have a collection .. why would i ever want to add multiple cores to it .. i mean i am trying to think of a reason why it would make sense to do that. Thanks On Wed, Feb 8, 2012 at 4:41 PM, Mark Miller wrote: > > On Feb 8, 2012, at 5:26 PM, Adeel Qureshi wrote: > > > okay so after reading Bruno's blog post .. lets add slice to the mix as > > well .. so we have got collections, cores, shards, partitions and slices > :) > > .. > > Yeah - heh - this has bugged me, but we have not really all come down on > agreement of terminology here. I was a fan of using shard for each node and > slice for partition. Another couple of committers wanted partitions rather > than slice. Another says slice in code, shard for both in terminology and > use context... > > I'd even go for shards as partitions and replicas for every node in a > shard. But those fine points are still settling ;) > > > > > The whole point with cores is to be able to have different schemas on the > > same solr server instance. So how does that changes with collections .. > may > > be an example might help .. if I want to setup a solrcloud cluster with 2 > > cores (different schema) .. with each core having 2 shards (i m assuming > > shards are really partitions here, across multiple nodes in the cluster) > .. > > with one shard being the replica.. > > So this would mean you want to create 2 collections. Think of a collection > as a bunch of SolrCores that all share the same schema and config. > > So you would start up 2 nodes set to one collection and with numShards=1 > that will give you one shard hosted by two identical SolrCores, giving you > a replication factor. The full index will be in each of the two SolrCores. > > Then if you start another two nodes and specify a different collection > name, you will get the same thing, but distinct from your first collection > (although, if both collections have compatible shema/config you can still > search across them). > > > > > > > On Wed, Feb 8, 2012 at 11:35 AM, Mark Miller > wrote: > > > >> > >> On Feb 8, 2012, at 10:31 AM, Adeel Qureshi wrote: > >> > >>> I have been using solr for a while and have recently started getting > into > >>> solrcloud .. i am a bit confused with some of the concepts .. > >>> > >>> 1. what exactly is the relationship between a collection and the core > .. > >>> can a core has multiple collections in it .. in this case all > collections > >>> within this core will have the same schema .. and i am assuming all > >>> instances of collections within the core can be deployed on different > >> solr > >>> nodes to achieve distributed search .. > >>> or is it the other way around where a collection can have multiple > cores > >> > >> Currently, a core basically equals a replica of the index. > >> > >> So you might have a collection called collection1 - lets say it's 2 > shards > >> and each shard has a single replica: > >> > >> Collection1 > >> shard1 replica1 > >> shard1 replica2 > >> shard2 replica1 > >> shard2 replica2 > >> > >> Each of those replicas is a core. So a collection has multiple cores > >> basically. Also, each of those cores can be on a different machine. So > yes, > >> you have distributed indexing and distributed search. > >> > >>> > >>> 2. at some places it has been pointed out that solrcloud doesnt > actually > >>> supports replication .. but in the solrcloud wiki the second example is > >>> supposed to be for replication .. so does solrcloud at this point > >> supports > >>> automatic replication where as you add more servers it automatically > uses > >>> the additional servers as replicas > >> > >> SolrCloud doesn't support the old style Solr replication concept. It > does > >> however, handle replication - it's just all pretty much automatic and > >> behind the scenes - eg all the information about
Re: solr cloud concepts
okay so after reading Bruno's blog post .. lets add slice to the mix as well .. so we have got collections, cores, shards, partitions and slices :) .. The whole point with cores is to be able to have different schemas on the same solr server instance. So how does that changes with collections .. may be an example might help .. if I want to setup a solrcloud cluster with 2 cores (different schema) .. with each core having 2 shards (i m assuming shards are really partitions here, across multiple nodes in the cluster) .. with one shard being the replica.. On Wed, Feb 8, 2012 at 11:35 AM, Mark Miller wrote: > > On Feb 8, 2012, at 10:31 AM, Adeel Qureshi wrote: > > > I have been using solr for a while and have recently started getting into > > solrcloud .. i am a bit confused with some of the concepts .. > > > > 1. what exactly is the relationship between a collection and the core .. > > can a core has multiple collections in it .. in this case all collections > > within this core will have the same schema .. and i am assuming all > > instances of collections within the core can be deployed on different > solr > > nodes to achieve distributed search .. > > or is it the other way around where a collection can have multiple cores > > Currently, a core basically equals a replica of the index. > > So you might have a collection called collection1 - lets say it's 2 shards > and each shard has a single replica: > > Collection1 > shard1 replica1 > shard1 replica2 > shard2 replica1 > shard2 replica2 > > Each of those replicas is a core. So a collection has multiple cores > basically. Also, each of those cores can be on a different machine. So yes, > you have distributed indexing and distributed search. > > > > > 2. at some places it has been pointed out that solrcloud doesnt actually > > supports replication .. but in the solrcloud wiki the second example is > > supposed to be for replication .. so does solrcloud at this point > supports > > automatic replication where as you add more servers it automatically uses > > the additional servers as replicas > > SolrCloud doesn't support the old style Solr replication concept. It does > however, handle replication - it's just all pretty much automatic and > behind the scenes - eg all the information about Solr replication in the > wiki documentation for previous versions of Solr is really not applicable. > We now achieve replica copies by sending documents to each shard one > document at a time so that we can support near realtime search. The old > style replication is only used in recovery, or when you start a new replica > machine and it has to 'catchup' to the other replicas. > > > > > I have a few more questions but I wanted to get these basic ones out of > the > > way first .. I would appreciate any response. > > Fire away. > > > > > Thanks > > Adeel > > - Mark Miller > lucidimagination.com > > > > > > > > > > > >
Re: data import in 4.0
Any comments .. please
I am able to do the bulkimport without nested query but with nested query it
just keeps working on it and never seems to end ..
I would appreciate any help
Thanks
Adeel
On Sat, Oct 22, 2011 at 11:12 AM, Adeel Qureshi wrote:
> yup that was it .. my data import files version was not the same as solr
> war .. now I am having another problem though
>
> I tried doing a simple data import
>
>
>
>
>
>
>
>
>
> simple in terms of just pulling up three fields from a table and adding to
> index and this worked fine but when I add a nested or joined table ..
>
>
>
>
>
>
>
>
>
>
>
>
> this data import doesnt seems to end .. it just keeps going .. i only have
> about 15000 records in the main table and about 22000 in the joined table ..
> but the Fetch count in dataimport handler status indicator thing shows that
> it has fetched close to half a million records or something .. i m not sure
> what those records are .. is there a way to see exactly what queries are
> being run by dataimport handler .. is there something wrong with my nested
> query ..
>
> THanks
> Adeel
>
>
> On Fri, Oct 21, 2011 at 3:05 PM, Alireza Salimi
> wrote:
>
>> So to me it heightens the probability of classloader conflicts,
>> I haven't worked with Solr 4.0, so I don't know if set of JAR files
>> are the same with Solr 3.4. Anyway, make sure that there is only
>> ONE instance of apache-solr-dataimporthandler-***.jar in your
>> whole tomcat+webapp.
>>
>> Maybe you have this jar file in CATALINA_HOME\lib folder.
>>
>> On Fri, Oct 21, 2011 at 3:06 PM, Adeel Qureshi > >wrote:
>>
>> > its deployed on a tomcat server ..
>> >
>> > On Fri, Oct 21, 2011 at 12:49 PM, Alireza Salimi
>> > wrote:
>> >
>> > > Hi,
>> > >
>> > > How do you start Solr, through start.jar or you deploy it to a web
>> > > container?
>> > > Sometimes problems like this are because of different class loaders.
>> > > I hope my answer would help you.
>> > >
>> > > Regards
>> > >
>> > >
>> > > On Fri, Oct 21, 2011 at 12:47 PM, Adeel Qureshi <
>> adeelmahm...@gmail.com
>> > > >wrote:
>> > >
>> > > > Hi I am trying to setup the data import handler with solr 4.0 and
>> > having
>> > > > some unexpected problems. I have a multi-core setup and only one
>> core
>> > > > needed
>> > > > the dataimport handler so I have added the request handler to it and
>> > > added
>> > > > the lib imports in config file
>> > > >
>> > > > > regex="apache-solr-dataimporthandler-\d.*\.jar"
>> > />
>> > > > > > > > regex="apache-solr-dataimporthandler-extras-\d.*\.jar" />
>> > > >
>> > > > for some reason this doesnt works .. it still keeps giving me
>> > > ClassNoFound
>> > > > error message so I moved the jars files to the shared lib folder and
>> > then
>> > > > atleast I was able to see the admin screen with the dataimport
>> plugin
>> > > > loaded. But when I try to do the import its throwing this error
>> message
>> > > >
>> > > > INFO: Starting Full Import
>> > > > Oct 21, 2011 11:35:41 AM org.apache.solr.core.SolrCore execute
>> > > > INFO: [DW] webapp=/solr path=/select
>> > > params={command=status&qt=/dataimport}
>> > > > status=0 QTime=0
>> > > > Oct 21, 2011 11:35:41 AM
>> org.apache.solr.handler.dataimport.SolrWriter
>> > > > readIndexerProperties
>> > > > WARNING: Unable to read: dataimport.properties
>> > > > Oct 21, 2011 11:35:41 AM
>> > org.apache.solr.handler.dataimport.DataImporter
>> > > > doFullImport
>> > > > SEVERE: Full Import failed
>> > > > java.lang.NoSuchMethodError:
>> > org.apache.solr.update.DeleteUpdateCommand:
>> > > > method ()V not found
>> > > >at
>> > > >
>> > > >
>> > >
>> >
>> org.apache.solr.handler.dataimport.SolrWriter.doDeleteAll(SolrWriter.java:193)
>> > > >at
>> > > >
>> > > >
>> > >
>> >
&g
Re: data import in 4.0
yup that was it .. my data import files version was not the same as solr war
.. now I am having another problem though
I tried doing a simple data import
simple in terms of just pulling up three fields from a table and adding to
index and this worked fine but when I add a nested or joined table ..
this data import doesnt seems to end .. it just keeps going .. i only have
about 15000 records in the main table and about 22000 in the joined table ..
but the Fetch count in dataimport handler status indicator thing shows that
it has fetched close to half a million records or something .. i m not sure
what those records are .. is there a way to see exactly what queries are
being run by dataimport handler .. is there something wrong with my nested
query ..
THanks
Adeel
On Fri, Oct 21, 2011 at 3:05 PM, Alireza Salimi wrote:
> So to me it heightens the probability of classloader conflicts,
> I haven't worked with Solr 4.0, so I don't know if set of JAR files
> are the same with Solr 3.4. Anyway, make sure that there is only
> ONE instance of apache-solr-dataimporthandler-***.jar in your
> whole tomcat+webapp.
>
> Maybe you have this jar file in CATALINA_HOME\lib folder.
>
> On Fri, Oct 21, 2011 at 3:06 PM, Adeel Qureshi >wrote:
>
> > its deployed on a tomcat server ..
> >
> > On Fri, Oct 21, 2011 at 12:49 PM, Alireza Salimi
> > wrote:
> >
> > > Hi,
> > >
> > > How do you start Solr, through start.jar or you deploy it to a web
> > > container?
> > > Sometimes problems like this are because of different class loaders.
> > > I hope my answer would help you.
> > >
> > > Regards
> > >
> > >
> > > On Fri, Oct 21, 2011 at 12:47 PM, Adeel Qureshi <
> adeelmahm...@gmail.com
> > > >wrote:
> > >
> > > > Hi I am trying to setup the data import handler with solr 4.0 and
> > having
> > > > some unexpected problems. I have a multi-core setup and only one core
> > > > needed
> > > > the dataimport handler so I have added the request handler to it and
> > > added
> > > > the lib imports in config file
> > > >
> > > > regex="apache-solr-dataimporthandler-\d.*\.jar"
> > />
> > > > > > > regex="apache-solr-dataimporthandler-extras-\d.*\.jar" />
> > > >
> > > > for some reason this doesnt works .. it still keeps giving me
> > > ClassNoFound
> > > > error message so I moved the jars files to the shared lib folder and
> > then
> > > > atleast I was able to see the admin screen with the dataimport plugin
> > > > loaded. But when I try to do the import its throwing this error
> message
> > > >
> > > > INFO: Starting Full Import
> > > > Oct 21, 2011 11:35:41 AM org.apache.solr.core.SolrCore execute
> > > > INFO: [DW] webapp=/solr path=/select
> > > params={command=status&qt=/dataimport}
> > > > status=0 QTime=0
> > > > Oct 21, 2011 11:35:41 AM
> org.apache.solr.handler.dataimport.SolrWriter
> > > > readIndexerProperties
> > > > WARNING: Unable to read: dataimport.properties
> > > > Oct 21, 2011 11:35:41 AM
> > org.apache.solr.handler.dataimport.DataImporter
> > > > doFullImport
> > > > SEVERE: Full Import failed
> > > > java.lang.NoSuchMethodError:
> > org.apache.solr.update.DeleteUpdateCommand:
> > > > method ()V not found
> > > >at
> > > >
> > > >
> > >
> >
> org.apache.solr.handler.dataimport.SolrWriter.doDeleteAll(SolrWriter.java:193)
> > > >at
> > > >
> > > >
> > >
> >
> org.apache.solr.handler.dataimport.DocBuilder.cleanByQuery(DocBuilder.java:1012)
> > > >at
> > > >
> > >
> >
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:183)
> > > >at
> > > >
> > > >
> > >
> >
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:335)
> > > >at
> > > >
> > > >
> > >
> >
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:393)
> > > >at
> > > >
> > > >
> > >
> >
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:374)
> > > > Oct 21, 2011 11:35:41 AM
> org.apache.solr.handler.dataimport.SolrWriter
>
Re: data import in 4.0
its deployed on a tomcat server ..
On Fri, Oct 21, 2011 at 12:49 PM, Alireza Salimi
wrote:
> Hi,
>
> How do you start Solr, through start.jar or you deploy it to a web
> container?
> Sometimes problems like this are because of different class loaders.
> I hope my answer would help you.
>
> Regards
>
>
> On Fri, Oct 21, 2011 at 12:47 PM, Adeel Qureshi >wrote:
>
> > Hi I am trying to setup the data import handler with solr 4.0 and having
> > some unexpected problems. I have a multi-core setup and only one core
> > needed
> > the dataimport handler so I have added the request handler to it and
> added
> > the lib imports in config file
> >
> >
> > > regex="apache-solr-dataimporthandler-extras-\d.*\.jar" />
> >
> > for some reason this doesnt works .. it still keeps giving me
> ClassNoFound
> > error message so I moved the jars files to the shared lib folder and then
> > atleast I was able to see the admin screen with the dataimport plugin
> > loaded. But when I try to do the import its throwing this error message
> >
> > INFO: Starting Full Import
> > Oct 21, 2011 11:35:41 AM org.apache.solr.core.SolrCore execute
> > INFO: [DW] webapp=/solr path=/select
> params={command=status&qt=/dataimport}
> > status=0 QTime=0
> > Oct 21, 2011 11:35:41 AM org.apache.solr.handler.dataimport.SolrWriter
> > readIndexerProperties
> > WARNING: Unable to read: dataimport.properties
> > Oct 21, 2011 11:35:41 AM org.apache.solr.handler.dataimport.DataImporter
> > doFullImport
> > SEVERE: Full Import failed
> > java.lang.NoSuchMethodError: org.apache.solr.update.DeleteUpdateCommand:
> > method ()V not found
> >at
> >
> >
> org.apache.solr.handler.dataimport.SolrWriter.doDeleteAll(SolrWriter.java:193)
> >at
> >
> >
> org.apache.solr.handler.dataimport.DocBuilder.cleanByQuery(DocBuilder.java:1012)
> >at
> >
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:183)
> >at
> >
> >
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:335)
> >at
> >
> >
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:393)
> >at
> >
> >
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:374)
> > Oct 21, 2011 11:35:41 AM org.apache.solr.handler.dataimport.SolrWriter
> > rollback
> > SEVERE: Exception while solr rollback.
> > java.lang.NoSuchMethodError:
> org.apache.solr.update.RollbackUpdateCommand:
> > method ()V not found
> >at
> >
> org.apache.solr.handler.dataimport.SolrWriter.rollback(SolrWriter.java:184)
> >at
> >
> org.apache.solr.handler.dataimport.DocBuilder.rollback(DocBuilder.java:249)
> >at
> >
> >
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:340)
> >at
> >
> >
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:393)
> >at
> >
> >
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:374)
> > Oct 21, 2011 11:35:43 AM org.apache.solr.core.SolrCore execute
> > INFO: [DW] webapp=/solr path=/select
> params={command=status&qt=/dataimport}
> > status=0 QTime=0
> >
> > Any ideas whats going on .. its complaining about a missing method in
> > dataimport classes which doesnt makes sense. Is this some kind of version
> > mismatch or what is going on.
> >
> > I would appreciate any comments.
> > Thanks
> > Adeel
> >
>
>
>
> --
> Alireza Salimi
> Java EE Developer
>
data import in 4.0
Hi I am trying to setup the data import handler with solr 4.0 and having
some unexpected problems. I have a multi-core setup and only one core needed
the dataimport handler so I have added the request handler to it and added
the lib imports in config file
for some reason this doesnt works .. it still keeps giving me ClassNoFound
error message so I moved the jars files to the shared lib folder and then
atleast I was able to see the admin screen with the dataimport plugin
loaded. But when I try to do the import its throwing this error message
INFO: Starting Full Import
Oct 21, 2011 11:35:41 AM org.apache.solr.core.SolrCore execute
INFO: [DW] webapp=/solr path=/select params={command=status&qt=/dataimport}
status=0 QTime=0
Oct 21, 2011 11:35:41 AM org.apache.solr.handler.dataimport.SolrWriter
readIndexerProperties
WARNING: Unable to read: dataimport.properties
Oct 21, 2011 11:35:41 AM org.apache.solr.handler.dataimport.DataImporter
doFullImport
SEVERE: Full Import failed
java.lang.NoSuchMethodError: org.apache.solr.update.DeleteUpdateCommand:
method ()V not found
at
org.apache.solr.handler.dataimport.SolrWriter.doDeleteAll(SolrWriter.java:193)
at
org.apache.solr.handler.dataimport.DocBuilder.cleanByQuery(DocBuilder.java:1012)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:183)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:335)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:393)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:374)
Oct 21, 2011 11:35:41 AM org.apache.solr.handler.dataimport.SolrWriter
rollback
SEVERE: Exception while solr rollback.
java.lang.NoSuchMethodError: org.apache.solr.update.RollbackUpdateCommand:
method ()V not found
at
org.apache.solr.handler.dataimport.SolrWriter.rollback(SolrWriter.java:184)
at
org.apache.solr.handler.dataimport.DocBuilder.rollback(DocBuilder.java:249)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:340)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:393)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:374)
Oct 21, 2011 11:35:43 AM org.apache.solr.core.SolrCore execute
INFO: [DW] webapp=/solr path=/select params={command=status&qt=/dataimport}
status=0 QTime=0
Any ideas whats going on .. its complaining about a missing method in
dataimport classes which doesnt makes sense. Is this some kind of version
mismatch or what is going on.
I would appreciate any comments.
Thanks
Adeel
schema design question
Hi there I have a question regarding how to setup schema for some data. This data is basically parent-child data for different types of records .. so a bunch of records representing projects and subprojects where each subproject has a parent project .. and a project has many child sub projects another bunch of records reprensenting data for projects and linked projects .. same parent child relationship here another bunch representing project and linked people .. so there are two ways I was thinking this kind of data can be indexed 1. create a single store called lets say CollectionData. use dynamic fields to post all this different data but use a type field to identify the type of records . e.g. to post two docs one representing project->linkedproject and another project->linkedpeople info 123 LinkedProjects child project name child project status ... parent info ... 123 LinkedPeople child person name ... parent info ... now from the same store I can run queries to get the different data while restricting the resultset on one type of records using the fq param .. 2. approach would be to create multiple stores for each different type of records .. with pretty much the same schema but now we dont need the type field because linkedProjects are in a linkedProjects store and linkedPeople are in linkedPeople store .. only drawback i guess is that you could have a few stores my question to you guys is which approach makes more sense. I would appreciate any comments. Thanks Adeel
updating documents while keeping unspecified fields
What I am trying to do is to update a document information while keeping data for the fields that arent being specified in the update. So e.g. if this is the schema 123 some title active if i send 123 closed it should update the status to be closed for this document but not wipe out title since it wasnt provided in the updated data. Is that possible by using some flags or something ??? Thanks Adeel
Re: SOLR-792 (hierarchical faceting) issue when only 1 document should be present in the pivot
I have had the same problem .. my facet pivots was returning results
something like
Cat-A (3)
Item X
Item Y
only 2 items instead of 3
or even
Cat-B (2)
no items
zero items instead of 2
so the parent level count didnt matched with the returned child pivots ..
but once I set the facet.pivot.mincount = 0 .. then it works fine .. is this
a bug or the desired behavior ..
On Wed, Nov 24, 2010 at 7:49 AM, Nicolas Peeters wrote:
> Hi Solr-Users,
>
> I realized that I can get the behaviour that I expect if I put
> facet.pivot.mincount to 0. However, I'm still puzzled why this needs to be
> 0
> and not 1. There's one occurence for this document, isn't it?
> With this value to 1, the print out of the pivot looks like this (where you
> clearly see (1) for "Value_that_can't_be_matched"):
>
> PIVOT: level1_loc_s,level2_loc_s,level3_loc_s
> level1_loc_s=Greater London (8)
> level2_loc_s=London (5)
>level3_loc_s=Mayfair (3)
>level3_loc_s=Hammersmith (2)
> level2_loc_s=Greenwich (3)
>level3_loc_s=Greenwich Centre (2)
> level3_loc_s=Value_that_cant_be_matched (1)
> level1_loc_s=Groot Amsterdam (5)
> level2_loc_s=Amsterdam (3)
>level3_loc_s=Jordaan (2)
> level3_loc_s=Centrum (1)
> level2_loc_s=Amstelveen (2)
>level3_loc_s=Centrum (2)
>
> Any expert advice on why this is the case is more than welcome!
>
> Best regards,
>
> Nicolas
>
> On Wed, Nov 24, 2010 at 2:27 PM, Nicolas Peeters >wrote:
>
> > Hi Solr Community,
> >
> > I've been experimenting with Solr 4.0 (trunk) in order to test the
> SOLR-792
> > feature. I have written a test that shows what I'm trying to ask.
> Basically,
> > I'm creating a hierarchy of the area/city/neighbourhood. The problem that
> I
> > see is that for documents that have only 1 item in a particular hierarchy
> > (e.g. Greater London/Greenwich/Centre (which I've called
> > "Value_that_cant_be_matched in this example"...)), these are not found by
> > the pivot facet. If I add a second one, then it works. I'm puzzled why
> this
> > is the case.
> >
> > This is the result of the Sytem.out that prints out the pivot facet
> fields
> > hierarchy (see line 86)
> >
> > PIVOT: level1_loc_s,level2_loc_s,level3_loc_s
> > level1_loc_s=Greater London (8)
> > level2_loc_s=London (5)
> > level3_loc_s=Mayfair (3)
> > level3_loc_s=Hammersmith (2)
> > level2_loc_s=Greenwich (3)
> > level3_loc_s=Greenwich Centre (2)
> > //--> why isn't there a
> > "level3_loc_s=Value_that_cant_be_matched (1)" here?
> > level1_loc_s=Groot Amsterdam (5)
> > level2_loc_s=Amsterdam (3)
> > level3_loc_s=Jordaan (2)
> > level2_loc_s=Amstelveen (2)
> > level3_loc_s=Centrum (2)
> >
> >
> > How can I make sure that Solr would find in the tree the single document
> > when I facet on this "location" hierarchy?
> >
> > Thank you very much for your help.
> >
> > Nicolas
> >
> > import java.io.IOException;
> > import java.net.MalformedURLException;
> > import java.util.ArrayList;
> > import java.util.List;
> > import java.util.Map;
> >
> > import org.apache.solr.client.solrj.SolrQuery;
> > import org.apache.solr.client.solrj.SolrServerException;
> > import org.apache.solr.client.solrj.impl.CommonsHttpSolrServer;
> > import org.apache.solr.client.solrj.response.PivotField;
> > import org.apache.solr.client.solrj.response.QueryResponse;
> > import org.apache.solr.common.SolrInputDocument;
> > import org.apache.solr.common.util.NamedList;
> > import org.junit.Assert;
> > import org.junit.Before;
> > import org.junit.Test;
> >
> > /**
> > * This is a test for hiearchical faceting based on SOLR-792 (I basically
> > just checkout the trunk of Solr-4.0).
> > *
> > * Unit test that shows the particular behaviour that I'm experiencing.
> > * I would have expected that the doc (see line 95) with as level3_loc_s
> > "Value_that_cant_be_matched" would appear in the pivot. It seems that you
> > actually need at least 2!
> > *
> > * @author npeeters
> > */
> > public class HierarchicalPivotTest {
> >
> > CommonsHttpSolrServer server;
> >
> > @Before
> > public void setup() throws MalformedURLException {
> > // the instance can be reused
> > this.server = new CommonsHttpSolrServer("
> > http://localhost:8983/solr";);
> > this.server.setSoTimeout(500); // socket read timeout
> > this.server.setConnectionTimeout(100);
> > this.server.setDefaultMaxConnectionsPerHost(100);
> > this.server.setMaxTotalConnections(100);
> > this.server.setFollowRedirects(false); // defaults to false
> > // allowCompression defaults to false.
> > }
> >
> > protected List createHierarchicalOrgData() {
> > int id = 1;
> > List docs = new
> ArrayList();
> > docs.add(makeTestDoc("id", id++, "name", "Organization " + id,
> > "level1_loc_s", "Groot Amsterdam", "level2_loc_s", "Amsterdam",
> > "level3_loc_s", "Centrum"));
> > docs.add(makeTestDoc("id", id++, "name
Re: setting up solr in eclipse
never mind .. re-did all the steps with a newer version of eclipse and jdk
.. and its all looking good now ..
On Fri, Dec 17, 2010 at 6:19 PM, Adeel Qureshi wrote:
> This is definitely a solr-dev mailing list question but I cant get the darn
> list to let me subscribe .. have subscribed several times but cant get the
> confirmation email ..so i am gonna post this here
>
> i am trying to setup solr/lucene trunk code in eclipse and after like 3
> days of figuring out things and looking up stuff and adding jars and linking
> folder and all that kind of stuff .. I am down to 20 errors .. and they
> seems to be of 2 types
>
> 1. there is noggit jar included in solr libs and its needed for json
> parsing classes .. so I downloaded noggit-1.0-SNAPSHOT.jar .. which is the
> only version I could find for it .. but it doesnt have some of the methods
> that are being used in solr e.g. JSONParser.getEventString .. does anyone
> knows where I can get an updated version of noggit jar
>
> 2. this is some weird error that I am getting .. a bunch of them but seem
> similar ..
> e.g.
> The method compare(SolrTestCaseJ4.Doc, SolrTestCaseJ4.Doc) of type new
> Comparator(){} must override a superclass method
> SolrTestCaseJ4.java/solr/src/test/org/apache/solrline 939Java
> Problem
> The method compare(TestGroupingSearch.Grp, TestGroupingSearch.Grp) of type
> new Comparator(){} must override a superclass
> methodTestGroupingSearch.java/solr/src/test/org/apache/solrline
> 547Java Problem
> any ideas what I am missing here ..
>
> I am using eclipse 3.5 with jdk 1.6 obviously (since its solr 4.0 code) ..
> I have included all the necessary jars .. except the correct version for
> noggit .. linked all the source folders mentioned in solr HowToContrib wiki
> ,, and applied the project and classpath files from it too ..
>
> Thanks
> Adeel
>
>
setting up solr in eclipse
This is definitely a solr-dev mailing list question but I cant get the darn
list to let me subscribe .. have subscribed several times but cant get the
confirmation email ..so i am gonna post this here
i am trying to setup solr/lucene trunk code in eclipse and after like 3 days
of figuring out things and looking up stuff and adding jars and linking
folder and all that kind of stuff .. I am down to 20 errors .. and they
seems to be of 2 types
1. there is noggit jar included in solr libs and its needed for json parsing
classes .. so I downloaded noggit-1.0-SNAPSHOT.jar .. which is the only
version I could find for it .. but it doesnt have some of the methods that
are being used in solr e.g. JSONParser.getEventString .. does anyone knows
where I can get an updated version of noggit jar
2. this is some weird error that I am getting .. a bunch of them but seem
similar ..
e.g.
The method compare(SolrTestCaseJ4.Doc, SolrTestCaseJ4.Doc) of type new
Comparator(){} must override a superclass method
SolrTestCaseJ4.java/solr/src/test/org/apache/solrline 939Java
Problem
The method compare(TestGroupingSearch.Grp, TestGroupingSearch.Grp) of type
new Comparator(){} must override a superclass
methodTestGroupingSearch.java/solr/src/test/org/apache/solrline
547Java Problem
any ideas what I am missing here ..
I am using eclipse 3.5 with jdk 1.6 obviously (since its solr 4.0 code) .. I
have included all the necessary jars .. except the correct version for
noggit .. linked all the source folders mentioned in solr HowToContrib wiki
,, and applied the project and classpath files from it too ..
Thanks
Adeel
Re: facet.pivot for date fields
i guess one last call for help .. i am assuming for people who wrote or have used the pivot faceting .. this should be a yes no question .. are date fields supported ? On Wed, Dec 15, 2010 at 12:58 PM, Adeel Qureshi wrote: > Thanks Pankaj - that was useful to know. I havent used the query stuff > before for facets .. so that was good to know .. but the problem is still > there because I want the hierarchical counts which is exactly what > facet.pivot does .. > > so e.g. i want to count for fieldC within fieldB and even fieldB within > fieldA .. that kind of stuff .. for string based fields .. facet.pivot does > exactly that and does it very well .. but it doesnt seems to work for date > ranges .. so in this case I want counts to be broken down by fieldA and > fieldB and then fieldB counts for monthly ranges .. I understand that I > might be able to use facet.query to construct several queries to get these > counts .. e.g. *facet.query=fieldA:someValue AND fieldB:someValue AND > fieldC:[NOW-1YEAR TO NOW]* .. but there could be thousand of possible > combinations for fieldA and fieldB which will require as many facet.queries > which I am assuming is not the way to go .. > > it might be confusing what I have explained above so the simple question > still is if there is a way to get date range counts included in facet.pivot > > Adeel > > > > On Tue, Dec 14, 2010 at 10:53 PM, pankaj bhatt wrote: > >> Hi Adeel, >> You can make use of facet.query attribute to make the Faceting work >> across a range of dates. Here i am using the duration, just replace the >> field with a field date and Range values as the DATE in SOLR Format. >> so your query parameter will be like this ( you can pass multiple >> parameter >> of "facet.query" name) >> >> http//blasdsdfsd/q?=asdfasd&facet.query=itemduration:[0 To >> 49]&facet.query=itemduration:[50 To 99]&facet.query=itemduration:[100 To >> 149] >> >> Hope, it helps. >> >> / Pankaj Bhatt. >> >> On Wed, Dec 15, 2010 at 2:01 AM, Adeel Qureshi > >wrote: >> >> > It doesnt seems like pivot facetting works on dates .. I was just >> curious >> > if >> > thats how its supposed to be or I am doing something wrong .. if I >> include >> > a >> > datefield in the pivot list .. i simply dont get any facet results back >> for >> > that datefield >> > >> > Thanks >> > Adeel >> > >> > >
Re: facet.pivot for date fields
Thanks Pankaj - that was useful to know. I havent used the query stuff before for facets .. so that was good to know .. but the problem is still there because I want the hierarchical counts which is exactly what facet.pivot does .. so e.g. i want to count for fieldC within fieldB and even fieldB within fieldA .. that kind of stuff .. for string based fields .. facet.pivot does exactly that and does it very well .. but it doesnt seems to work for date ranges .. so in this case I want counts to be broken down by fieldA and fieldB and then fieldB counts for monthly ranges .. I understand that I might be able to use facet.query to construct several queries to get these counts .. e.g. *facet.query=fieldA:someValue AND fieldB:someValue AND fieldC:[NOW-1YEAR TO NOW]* .. but there could be thousand of possible combinations for fieldA and fieldB which will require as many facet.queries which I am assuming is not the way to go .. it might be confusing what I have explained above so the simple question still is if there is a way to get date range counts included in facet.pivot Adeel On Tue, Dec 14, 2010 at 10:53 PM, pankaj bhatt wrote: > Hi Adeel, > You can make use of facet.query attribute to make the Faceting work > across a range of dates. Here i am using the duration, just replace the > field with a field date and Range values as the DATE in SOLR Format. > so your query parameter will be like this ( you can pass multiple parameter > of "facet.query" name) > > http//blasdsdfsd/q?=asdfasd&facet.query=itemduration:[0 To > 49]&facet.query=itemduration:[50 To 99]&facet.query=itemduration:[100 To > 149] > > Hope, it helps. > > / Pankaj Bhatt. > > On Wed, Dec 15, 2010 at 2:01 AM, Adeel Qureshi >wrote: > > > It doesnt seems like pivot facetting works on dates .. I was just curious > > if > > thats how its supposed to be or I am doing something wrong .. if I > include > > a > > datefield in the pivot list .. i simply dont get any facet results back > for > > that datefield > > > > Thanks > > Adeel > > >
facet.pivot for date fields
It doesnt seems like pivot facetting works on dates .. I was just curious if thats how its supposed to be or I am doing something wrong .. if I include a datefield in the pivot list .. i simply dont get any facet results back for that datefield Thanks Adeel
Re: Solr Reporting
Hi Peter I understand what you are saying but I think you are thinking more of report as graph and analysis and summary kind of data .. for my reports I do need to include all records that qualify certain criteria .. e.g. a listing of all orders placed in last 6 months .. now that could be 1 orders and yes I will need probably a report that summarizes all that data but at the same time .. I need all those 1 records to be exported in an excel file .. those are the reports that I am talking about .. and 3 probably is a stretch .. it might be 10-15000 at the most but I guess its still the same idea .. and yes I realize that its alot of data to be transferred over http .. but thats exactly why i am asking for suggestion on how to do .. I find it hard to believe that this is an unusual requirement .. I think most companies do reports that dump all records from databases in excel files .. so again to clarify I definitely need reports that present statistics and averages and yes I will be using facets and all kind of stuff there and I am not so concerned about those reports because like you pointed out, for those reports there will be very little data transfer but its the full data dump reports that I am trying to figure out the best way to handle. Thanks for your help Adeel On Thu, Sep 23, 2010 at 11:43 AM, Peter Sturge wrote: > Hi, > > Are you going to generate a report with 3 records in it? That will > be a very large report - will anyone really want to read through that? > If you want/need 'summary' reports - i.e. stats on on the 30k records, > it is much more efficient to setup faceting and/or server-side > analysis to do this, rather than download > 3 records to a client, then do statistical analysis on the result. > It will take a while to stream 3 records over an http connection, > and, if you're building, say, a PDF table for 30k records, that will > take some time as well. > Server-side analysis then just send the results will work better, if > that fits your remit for reporting. > > Peter > > > > On Thu, Sep 23, 2010 at 4:14 PM, Adeel Qureshi > wrote: > > Thank you for your suggestions .. makes sense and I didnt knew about the > > XsltResponseWriter .. that opens up door to all kind of possibilities > ..so > > its great to know about that > > > > but before I go that route .. what about performance .. In Solr Wiki it > > mentions that XSLT transformation isnt so bad in terms of memory usage > but I > > guess its all relative to the amount of data and obviously system > resources > > .. > > > > my data set will be around 15000 - 30'000 records at the most ..I do have > > about 30 some fields but all fields are either small strings (less than > 500 > > chars) or dates, int, booleans etc .. so should I be worried about > > performances problems while doing the XSLT translations .. secondly for > > reports Ill have to request solr to send all 15000 some records at the > same > > time to be entered in report output files .. is there a way to kind of > > stream that process .. well I think Solr native xml is already streamed > to > > you but sounds like for the translation it will have to load the whole > thing > > in RAM .. > > > > and again what about SolrJ .. isnt that supposed to provide better > > performance since its in java .. well I guess it shouldnt be much > different > > since it also uses the HTTP calls to communicate to Solr .. > > > > Thanks for your help > > Adeel > > > > On Thu, Sep 23, 2010 at 7:16 AM, kenf_nc > wrote: > > > >> > >> keep in mind that the paradigm isn't completely useless, > >> the > >> str is a data type (string), it can be int, float, double, date, and > >> others. > >> So to not lose any information you may want to do something like: > >> > >> 123 > >> xyz > >> > >> Which I agree makes more sense to me. The name of the field is more > >> important than it's datatype, but I don't want to lose track of the data > >> type. > >> > >> Ken > >> -- > >> View this message in context: > >> > http://lucene.472066.n3.nabble.com/Solr-Reporting-tp1565271p1567604.html > >> Sent from the Solr - User mailing list archive at Nabble.com. > >> > > >
Re: Solr Reporting
Thank you for your suggestions .. makes sense and I didnt knew about the XsltResponseWriter .. that opens up door to all kind of possibilities ..so its great to know about that but before I go that route .. what about performance .. In Solr Wiki it mentions that XSLT transformation isnt so bad in terms of memory usage but I guess its all relative to the amount of data and obviously system resources .. my data set will be around 15000 - 30'000 records at the most ..I do have about 30 some fields but all fields are either small strings (less than 500 chars) or dates, int, booleans etc .. so should I be worried about performances problems while doing the XSLT translations .. secondly for reports Ill have to request solr to send all 15000 some records at the same time to be entered in report output files .. is there a way to kind of stream that process .. well I think Solr native xml is already streamed to you but sounds like for the translation it will have to load the whole thing in RAM .. and again what about SolrJ .. isnt that supposed to provide better performance since its in java .. well I guess it shouldnt be much different since it also uses the HTTP calls to communicate to Solr .. Thanks for your help Adeel On Thu, Sep 23, 2010 at 7:16 AM, kenf_nc wrote: > > keep in mind that the paradigm isn't completely useless, > the > str is a data type (string), it can be int, float, double, date, and > others. > So to not lose any information you may want to do something like: > > 123 > xyz > > Which I agree makes more sense to me. The name of the field is more > important than it's datatype, but I don't want to lose track of the data > type. > > Ken > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Solr-Reporting-tp1565271p1567604.html > Sent from the Solr - User mailing list archive at Nabble.com. >
Solr Reporting
This probably isnt directly a solr user type question but its close enough so I am gonna post it here. I have been using solr for a few months now and it works just out of this world so I definitely love the software (and obviously lucene too) .. but I feel that solr output xml is in kind of weird format .. I mean its in a format that simply makes it difficult to plug solr output xml in any xml reading tool or api .. this whole concept of using 123 instead of 123 doesnt makse sense to me .. what I am trying to do now is setup a reporting system off of solr .. and the concept is simply .. let the user do all the searches, facet etc and once they have finalized on some results .. simply allow them to export those results in an excel or pdf file .. what I have setup right now is I simply let the export feature use the same solr query that user used to search their results .. send that query to solr again and get all results back and simply iterate over xml and dump all data in an excel file this has worked fine in most situations but I want to improve this process and specifically use jasper reports for reporting .. and I want to use ireport to design my report templates .. thats where solr output xml format is causing problems .. as I cant figure out how to make it work with ireport because of solr xml not having any named nodes .. it all looks like the same nodes and ireport cant distinguish one column from another .. so I am thinking a couple of solutions here and wanted to get some suggestions from you guys on how to do it best 1. receive solr output xml .. convert it to a more readable xml form .. use named nodes instead of nodes by data type 123 xyz => 123 xyz and then feed that to jasper report template 2. use solrJ to recieve solr output in the NamedList resultset as it returns ..I havent tried this method so I am not sure how useful or easy to work, this NamedList structure is .. in this I would be feeding Collection of NamedList items to jasper .. havent played around with this so not sure how well its gonna work out .. if you have tried something like this please let me know how it worked out for u I would appreciate absolutely any kind of comments on this Thanks Adeel
