Re: Storing different entities in Solr
Thanks for the replies guys. I am not at work so I don't have the exact schema but here's what it roughly looks like: Request: == id client_id pm_id pm2_id title Advisor: == id person_id address_id bio sector (IT, doctor, etc) There's another table. RequestAdvisor: = id advisor_id request_id The idea of adding a prefix to primary keys does sound good. I can just do advisor_123 and request_12345. On Sun, May 30, 2010 at 9:22 PM, Bill Au bill.w...@gmail.com wrote: There is only one primary key in a single index. If the id of your different document types do collide, you can simply add a prefix or suffix to make them unique. Bill On Fri, May 28, 2010 at 1:12 PM, Moazzam Khan moazz...@gmail.com wrote: Thanks for all your answers guys. Requests and consultants have a many to many relationship so I can't store request info in a document with advisorID as the primary key. Bill's solution and multicore solutions might be what I am looking for. Bill, will I be able to have 2 primary keys (so I can update and delete documents)? If yes, can you please give me a link or someting where I can get more info on this? Thanks, Moazzam On Fri, May 28, 2010 at 11:50 AM, Bill Au bill.w...@gmail.com wrote: You can keep different type of documents in the same index. If each document has a type field. You can restrict your searches to specific type(s) of document by using a filter query, which is very fast and efficient. Bill On Fri, May 28, 2010 at 12:28 PM, Nagelberg, Kallin knagelb...@globeandmail.com wrote: Multi-core is an option, but keep in mind if you go that route you will need to do two searches to correlate data between the two. -Kallin Nagelberg -Original Message- From: Robert Zotter [mailto:robertzot...@gmail.com] Sent: Friday, May 28, 2010 12:26 PM To: solr-user@lucene.apache.org Subject: Re: Storing different entities in Solr Sounds like you'll want to use a multiple core setup. One core fore each type of document http://wiki.apache.org/solr/CoreAdmin -- View this message in context: http://lucene.472066.n3.nabble.com/Storing-different-entities-in-Solr-tp852299p852346.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Storing different entities in Solr
There is only one primary key in a single index. If the id of your different document types do collide, you can simply add a prefix or suffix to make them unique. Bill On Fri, May 28, 2010 at 1:12 PM, Moazzam Khan moazz...@gmail.com wrote: Thanks for all your answers guys. Requests and consultants have a many to many relationship so I can't store request info in a document with advisorID as the primary key. Bill's solution and multicore solutions might be what I am looking for. Bill, will I be able to have 2 primary keys (so I can update and delete documents)? If yes, can you please give me a link or someting where I can get more info on this? Thanks, Moazzam On Fri, May 28, 2010 at 11:50 AM, Bill Au bill.w...@gmail.com wrote: You can keep different type of documents in the same index. If each document has a type field. You can restrict your searches to specific type(s) of document by using a filter query, which is very fast and efficient. Bill On Fri, May 28, 2010 at 12:28 PM, Nagelberg, Kallin knagelb...@globeandmail.com wrote: Multi-core is an option, but keep in mind if you go that route you will need to do two searches to correlate data between the two. -Kallin Nagelberg -Original Message- From: Robert Zotter [mailto:robertzot...@gmail.com] Sent: Friday, May 28, 2010 12:26 PM To: solr-user@lucene.apache.org Subject: Re: Storing different entities in Solr Sounds like you'll want to use a multiple core setup. One core fore each type of document http://wiki.apache.org/solr/CoreAdmin -- View this message in context: http://lucene.472066.n3.nabble.com/Storing-different-entities-in-Solr-tp852299p852346.html Sent from the Solr - User mailing list archive at Nabble.com.
Storing different entities in Solr
Hi Guys, Is there a way to store 2 types of things in Solr. We have a list of consultants and a list of consultation requests. and I want to store them as separate documents. Can I do this with one instance of Solr or do I have to have two instances? Thanks, MOazzam
RE: Storing different entities in Solr
Good read here: http://mysolr.com/tips/denormalized-data-structure/ . Are consultation requests unique to each consultant? In that case you could represent the request as a Json String and store it as a multi-valued string field for each consultant, though that makes querying against requests trickier. If you need to search against specific fields in the consultant requests than you could try a schema where the consultant is your primary entity and have fields like consultantrequests-field1, consultantrequests-field2, consultantrequests-field3 and then one consultantrequests-fulljson all multi-valued. You could query against the specific fields, then associate to the whole request by searching the json object. It's an approach I've used with success. -Kallin Nagelberg -Original Message- From: Moazzam Khan [mailto:moazz...@gmail.com] Sent: Friday, May 28, 2010 12:17 PM To: solr-user@lucene.apache.org Subject: Storing different entities in Solr Hi Guys, Is there a way to store 2 types of things in Solr. We have a list of consultants and a list of consultation requests. and I want to store them as separate documents. Can I do this with one instance of Solr or do I have to have two instances? Thanks, MOazzam
Re: Storing different entities in Solr
Sounds like you'll want to use a multiple core setup. One core fore each type of document http://wiki.apache.org/solr/CoreAdmin -- View this message in context: http://lucene.472066.n3.nabble.com/Storing-different-entities-in-Solr-tp852299p852346.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: Storing different entities in Solr
Multi-core is an option, but keep in mind if you go that route you will need to do two searches to correlate data between the two. -Kallin Nagelberg -Original Message- From: Robert Zotter [mailto:robertzot...@gmail.com] Sent: Friday, May 28, 2010 12:26 PM To: solr-user@lucene.apache.org Subject: Re: Storing different entities in Solr Sounds like you'll want to use a multiple core setup. One core fore each type of document http://wiki.apache.org/solr/CoreAdmin -- View this message in context: http://lucene.472066.n3.nabble.com/Storing-different-entities-in-Solr-tp852299p852346.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Storing different entities in Solr
You can keep different type of documents in the same index. If each document has a type field. You can restrict your searches to specific type(s) of document by using a filter query, which is very fast and efficient. Bill On Fri, May 28, 2010 at 12:28 PM, Nagelberg, Kallin knagelb...@globeandmail.com wrote: Multi-core is an option, but keep in mind if you go that route you will need to do two searches to correlate data between the two. -Kallin Nagelberg -Original Message- From: Robert Zotter [mailto:robertzot...@gmail.com] Sent: Friday, May 28, 2010 12:26 PM To: solr-user@lucene.apache.org Subject: Re: Storing different entities in Solr Sounds like you'll want to use a multiple core setup. One core fore each type of document http://wiki.apache.org/solr/CoreAdmin -- View this message in context: http://lucene.472066.n3.nabble.com/Storing-different-entities-in-Solr-tp852299p852346.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Storing different entities in Solr
Thanks for all your answers guys. Requests and consultants have a many to many relationship so I can't store request info in a document with advisorID as the primary key. Bill's solution and multicore solutions might be what I am looking for. Bill, will I be able to have 2 primary keys (so I can update and delete documents)? If yes, can you please give me a link or someting where I can get more info on this? Thanks, Moazzam On Fri, May 28, 2010 at 11:50 AM, Bill Au bill.w...@gmail.com wrote: You can keep different type of documents in the same index. If each document has a type field. You can restrict your searches to specific type(s) of document by using a filter query, which is very fast and efficient. Bill On Fri, May 28, 2010 at 12:28 PM, Nagelberg, Kallin knagelb...@globeandmail.com wrote: Multi-core is an option, but keep in mind if you go that route you will need to do two searches to correlate data between the two. -Kallin Nagelberg -Original Message- From: Robert Zotter [mailto:robertzot...@gmail.com] Sent: Friday, May 28, 2010 12:26 PM To: solr-user@lucene.apache.org Subject: Re: Storing different entities in Solr Sounds like you'll want to use a multiple core setup. One core fore each type of document http://wiki.apache.org/solr/CoreAdmin -- View this message in context: http://lucene.472066.n3.nabble.com/Storing-different-entities-in-Solr-tp852299p852346.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Storing different entities in Solr
Hi, So for your use case are you wanting to search for a consultant then look at all of his or her request or pull both at the same time? In both cases one index should suffice. In you define a primary key field and use it for both doc types it shouldn't be an issue. Unless your dataset in very large it would reduce the overhead of running a multicore solution especially in indexing etc David Stuart On 28 May 2010, at 18:12, Moazzam Khan moazz...@gmail.com wrote: Thanks for all your answers guys. Requests and consultants have a many to many relationship so I can't store request info in a document with advisorID as the primary key. Bill's solution and multicore solutions might be what I am looking for. Bill, will I be able to have 2 primary keys (so I can update and delete documents)? If yes, can you please give me a link or someting where I can get more info on this? Thanks, Moazzam On Fri, May 28, 2010 at 11:50 AM, Bill Au bill.w...@gmail.com wrote: You can keep different type of documents in the same index. If each document has a type field. You can restrict your searches to specific type(s) of document by using a filter query, which is very fast and efficient. Bill On Fri, May 28, 2010 at 12:28 PM, Nagelberg, Kallin knagelb...@globeandmail.com wrote: Multi-core is an option, but keep in mind if you go that route you will need to do two searches to correlate data between the two. -Kallin Nagelberg -Original Message- From: Robert Zotter [mailto:robertzot...@gmail.com] Sent: Friday, May 28, 2010 12:26 PM To: solr-user@lucene.apache.org Subject: Re: Storing different entities in Solr Sounds like you'll want to use a multiple core setup. One core fore each type of document http://wiki.apache.org/solr/CoreAdmin -- View this message in context: http://lucene.472066.n3.nabble.com/Storing-different-entities-in-Solr-tp852299p852346.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Storing different entities in Solr
You most certainly *can* store the many-many relationship, you are just denormalizing your data. I know it goes against the grain of any good database admin, but it's very often a good solution for a search application. You've gotta forget almost everything you learned about how data *should* be stored in databases when working with a search app. Well, perhaps I'm overstating a bit, but you get the idea When I see messages about primary keys and foreign keys etc, I break out in hives. It's almost always a mistake to try to force lucene/solr to behave like a database. Whenever you find yourself trying, stop, take a deep breath, and think about searching G... A lot depends on how much data we're talking about here. If fully denormalizing things would cost you 10M, who cares? If it would cost you 100G, it's a different story Best Erick On Fri, May 28, 2010 at 1:12 PM, Moazzam Khan moazz...@gmail.com wrote: Thanks for all your answers guys. Requests and consultants have a many to many relationship so I can't store request info in a document with advisorID as the primary key. Bill's solution and multicore solutions might be what I am looking for. Bill, will I be able to have 2 primary keys (so I can update and delete documents)? If yes, can you please give me a link or someting where I can get more info on this? Thanks, Moazzam On Fri, May 28, 2010 at 11:50 AM, Bill Au bill.w...@gmail.com wrote: You can keep different type of documents in the same index. If each document has a type field. You can restrict your searches to specific type(s) of document by using a filter query, which is very fast and efficient. Bill On Fri, May 28, 2010 at 12:28 PM, Nagelberg, Kallin knagelb...@globeandmail.com wrote: Multi-core is an option, but keep in mind if you go that route you will need to do two searches to correlate data between the two. -Kallin Nagelberg -Original Message- From: Robert Zotter [mailto:robertzot...@gmail.com] Sent: Friday, May 28, 2010 12:26 PM To: solr-user@lucene.apache.org Subject: Re: Storing different entities in Solr Sounds like you'll want to use a multiple core setup. One core fore each type of document http://wiki.apache.org/solr/CoreAdmin -- View this message in context: http://lucene.472066.n3.nabble.com/Storing-different-entities-in-Solr-tp852299p852346.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: Storing different entities in Solr
I agree with Erick, Could you show us what these two entities look like, and the total count of each? That might shed some light on the appropriate approach. -Kallin Nagelberg -Original Message- From: Erick Erickson [mailto:erickerick...@gmail.com] Sent: Friday, May 28, 2010 2:36 PM To: solr-user@lucene.apache.org Subject: Re: Storing different entities in Solr You most certainly *can* store the many-many relationship, you are just denormalizing your data. I know it goes against the grain of any good database admin, but it's very often a good solution for a search application. You've gotta forget almost everything you learned about how data *should* be stored in databases when working with a search app. Well, perhaps I'm overstating a bit, but you get the idea When I see messages about primary keys and foreign keys etc, I break out in hives. It's almost always a mistake to try to force lucene/solr to behave like a database. Whenever you find yourself trying, stop, take a deep breath, and think about searching G... A lot depends on how much data we're talking about here. If fully denormalizing things would cost you 10M, who cares? If it would cost you 100G, it's a different story Best Erick On Fri, May 28, 2010 at 1:12 PM, Moazzam Khan moazz...@gmail.com wrote: Thanks for all your answers guys. Requests and consultants have a many to many relationship so I can't store request info in a document with advisorID as the primary key. Bill's solution and multicore solutions might be what I am looking for. Bill, will I be able to have 2 primary keys (so I can update and delete documents)? If yes, can you please give me a link or someting where I can get more info on this? Thanks, Moazzam On Fri, May 28, 2010 at 11:50 AM, Bill Au bill.w...@gmail.com wrote: You can keep different type of documents in the same index. If each document has a type field. You can restrict your searches to specific type(s) of document by using a filter query, which is very fast and efficient. Bill On Fri, May 28, 2010 at 12:28 PM, Nagelberg, Kallin knagelb...@globeandmail.com wrote: Multi-core is an option, but keep in mind if you go that route you will need to do two searches to correlate data between the two. -Kallin Nagelberg -Original Message- From: Robert Zotter [mailto:robertzot...@gmail.com] Sent: Friday, May 28, 2010 12:26 PM To: solr-user@lucene.apache.org Subject: Re: Storing different entities in Solr Sounds like you'll want to use a multiple core setup. One core fore each type of document http://wiki.apache.org/solr/CoreAdmin -- View this message in context: http://lucene.472066.n3.nabble.com/Storing-different-entities-in-Solr-tp852299p852346.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Storing different entities in Solr
The size of the join table is the number of documents, if you denormalize the two tables. On Fri, May 28, 2010 at 11:38 AM, Nagelberg, Kallin knagelb...@globeandmail.com wrote: I agree with Erick, Could you show us what these two entities look like, and the total count of each? That might shed some light on the appropriate approach. -Kallin Nagelberg -Original Message- From: Erick Erickson [mailto:erickerick...@gmail.com] Sent: Friday, May 28, 2010 2:36 PM To: solr-user@lucene.apache.org Subject: Re: Storing different entities in Solr You most certainly *can* store the many-many relationship, you are just denormalizing your data. I know it goes against the grain of any good database admin, but it's very often a good solution for a search application. You've gotta forget almost everything you learned about how data *should* be stored in databases when working with a search app. Well, perhaps I'm overstating a bit, but you get the idea When I see messages about primary keys and foreign keys etc, I break out in hives. It's almost always a mistake to try to force lucene/solr to behave like a database. Whenever you find yourself trying, stop, take a deep breath, and think about searching G... A lot depends on how much data we're talking about here. If fully denormalizing things would cost you 10M, who cares? If it would cost you 100G, it's a different story Best Erick On Fri, May 28, 2010 at 1:12 PM, Moazzam Khan moazz...@gmail.com wrote: Thanks for all your answers guys. Requests and consultants have a many to many relationship so I can't store request info in a document with advisorID as the primary key. Bill's solution and multicore solutions might be what I am looking for. Bill, will I be able to have 2 primary keys (so I can update and delete documents)? If yes, can you please give me a link or someting where I can get more info on this? Thanks, Moazzam On Fri, May 28, 2010 at 11:50 AM, Bill Au bill.w...@gmail.com wrote: You can keep different type of documents in the same index. If each document has a type field. You can restrict your searches to specific type(s) of document by using a filter query, which is very fast and efficient. Bill On Fri, May 28, 2010 at 12:28 PM, Nagelberg, Kallin knagelb...@globeandmail.com wrote: Multi-core is an option, but keep in mind if you go that route you will need to do two searches to correlate data between the two. -Kallin Nagelberg -Original Message- From: Robert Zotter [mailto:robertzot...@gmail.com] Sent: Friday, May 28, 2010 12:26 PM To: solr-user@lucene.apache.org Subject: Re: Storing different entities in Solr Sounds like you'll want to use a multiple core setup. One core fore each type of document http://wiki.apache.org/solr/CoreAdmin -- View this message in context: http://lucene.472066.n3.nabble.com/Storing-different-entities-in-Solr-tp852299p852346.html Sent from the Solr - User mailing list archive at Nabble.com. -- Lance Norskog goks...@gmail.com
