On Mon, Feb 14, 2011 at 10:04 AM, Alex Cowell wrote:
> There seem to be some nuances which we have yet to encounter/discover like
> the way you've implemented the processCommit() method to wait for all the
> adds/deletes to complete before sending the commits. Are these things which
> you were awa
I've uploaded a patch of what we've done so far:
https://issues.apache.org/jira/browse/SOLR-2358
It's still very much work in progress and there are some obvious issues
which are being resolved at the moment (such as the inefficient method of
waiting for all the docs to be processed before distri
I haven't had time to follow all of this discussion, but this issue might help:
https://issues.apache.org/jira/browse/SOLR-2355
It's an implementation of the basic
http://localhost:8983/solr/update/csv?shards=shard1,shard2...
-Yonik
http://lucidimagination.com
On Mon, Feb 7, 2011 at 8:55 AM, Upa
Surely you want to be implementing an UpdateRequestProcessor,
rather than a RequestHandler.
The ContentStreamHandlerBase, in the handleRequestBody method
gets an UpdateRequestProcessor and uses it to process the
request. What we need is that handleRequestBody method to, as you
have suggested, chec
I'm saying that deterministic policies are a requirement that
*some* people will want. Others might want a random spread. Thus,
I'd have deterministic based on ID and random as the two initial
implementations.
Upayavira
NB. In case folks haven't worked it out already, I have been
tasked to mentor
Hey,
We're making good progress, but our DistributedUpdateRequestHandler is
having a bit of an identity crisis, so we thought we'd ask what other
people's opinions are. The current situation is as follows:
We've added a method to ContentStreamHandlerBase to check if an update
request is distribut
Hi
Good call about the policies being deterministic, should've thought of that
earlier.
We've changed the patch to include this and I've removed the random
assignment one (for obvious reasons).
Take a look and let me know what's to do. (
https://issues.apache.org/jira/browse/SOLR-2341)
Cheers
On Thu, 03 Feb 2011 15:12 +, "Alex Cowell"
wrote:
Hi all,
Just a couple of questions that have arisen.
1. For handling non-distributed update requests (shards param
is not present or is invalid), our code currently
* assumes the user would like the data indexed, so gets the
req
Hi all,
Just a couple of questions that have arisen.
1. For handling non-distributed update requests (shards param is not present
or is invalid), our code currently
- assumes the user would like the data indexed, so gets the request
handler assigned to "/update"
- executes the request u
On Tue, 01 Feb 2011 19:52 -0800, "Lance Norskog"
wrote:
> Another use case is that N indexers operate independently, all pulling
> data from the same database. Each has a separate query to get the
> documents in its policy.
But surely in this case, you are externalising the policy, and Solr
do
Another use case is that N indexers operate independently, all pulling
data from the same database. Each has a separate query to get the
documents in its policy.
On Tue, Feb 1, 2011 at 12:38 PM, Upayavira wrote:
>
> On Tue, 01 Feb 2011 19:04 +, "Alex Cowell" wrote:
>
> I noticed there is a
On Tue, 01 Feb 2011 19:04 +, "Alex Cowell"
wrote:
I noticed there is a comment in the
org.apache.solr.servlet.DirectSolrConnection class which
reads, "//Find a way to turn List into
File/SolrDocument". Did anyone find a way to do this?
Turns out that comment was left over from som
>
> I noticed there is a comment in the
> org.apache.solr.servlet.DirectSolrConnection class which reads, "//Find a
> way to turn List into File/SolrDocument". Did anyone find a
> way to do this?
>
Turns out that comment was left over from some experimenting one of our team
was doing. But I suppos
>
> Your code looks fine to me, except it should take in a SolrDocument
> object or list of, rather than strings. Then, for your Hash version, you
> can take a hash of the "id" field.
>
As far as I can see I have access to a List that
> represents all of the files being POSTed. Do I want to open t
Hello
Thanks for your prompt reply.
In regards to using a SolrDocument instead of Strings (and I agree
that List doesn't seem to be the best way of going) how do I
get reference to a SolrDoc?
As far as I can see I have access to a List that
represents all of the files being POSTed. Do I want to
On Tue, 01 Feb 2011 00:26 +, "William Mayor"
wrote:
> Hi Guys
>
> I've had a go at creating the ShardDistributionPolicy interface and a
> few implementations. I've created a patch
> (https://issues.apache.org/jira/browse/SOLR-2341) let me know what
> needs doing.
> Currently I assume that t
(I'm sending this on behalf of William, a guy on our team working on
ShardDistributedPolicy):
Hi Guys
I've had a go at creating the ShardDistributionPolicy interface and a
few implementations. I've created a patch
(https://issues.apache.org/jira/browse/SOLR-2341) let me know what
needs doing.
Cu
Hi Guys
I've had a go at creating the ShardDistributionPolicy interface and a
few implementations. I've created a patch
(https://issues.apache.org/jira/browse/SOLR-2341) let me know what
needs doing.
Currently I assume that the documents passed to the policy will be
represented by some kind of id
Lance,
Firstly, we're proposing a ShardDistributionPolicy interface for which
there is a default (mod of the doc ID) but other implementations are
possible. Another easy implementation would be a randomised or round
robin one.
As to threading, the first task would be to put all of the source
docu
I would suggest that a DistributedRequestUpdateHandler run
single-threaded, doing only one document at a time. If I want more
than one, I run it twice or N times with my own program.
Also, this should have a policy object which decides exactly how
documents are distributed. There are different tec
Hello Yonik,
On Thu, 2011-01-27 at 08:01 -0500, Yonik Seeley wrote:
> Making it easy for clients I think is key... one should be able to
> update any node in the solr cluster and have solr take care of the
> hard part about updating all relevant shards. This will most likely
> involve an update p
Hi Yonik and Upayavira,
Thank you both for your insightful responses. We now have a much better
understanding of how to implement distributed indexing, although no doubt
more issues will emerge along the way.
Just to clarify (and for critique), our approach goes something like this:
We will use a
On Fri, Jan 28, 2011 at 7:55 AM, Upayavira wrote:
>
> On Thu, 27 Jan 2011 16:01 +, "Alex Cowell" wrote:
>
> Making it easy for clients I think is key... one should be able to
> update any node in the solr cluster and have solr take care of the
> hard part about updating all relevant shards.
Another point that will need some thought, as I have heard alluded to,
is error handling.
Currently, as I understand it, if you post 500 documents to Solr, and
one has an error, the whole batch will fail.
Leaving aside whether that is the best behaviour, it is a behaviour that
will be impossible
On Thu, 27 Jan 2011 16:01 +, "Alex Cowell"
wrote:
Making it easy for clients I think is key... one should be
able to
update any node in the solr cluster and have solr take care of
the
hard part about updating all relevant shards. This will most
likely
involve an update process
Hi Soheb,
On Wed, 26 Jan 2011 16:29 +, "Soheb Mahmood"
wrote:
> We are going to implement distributed indexing for Solr - without the
> use of SolrCloud (so it can be easily up-scaled). We have a deadline by
> February to get this done, so we need to get cracking ;)
:-)
> So far, we've h
>
> Making it easy for clients I think is key... one should be able to
> update any node in the solr cluster and have solr take care of the
> hard part about updating all relevant shards. This will most likely
> involve an update processor. This approach allows all existing update
> methods (incl
On Wed, Jan 26, 2011 at 11:29 AM, Soheb Mahmood wrote:
> We were wondering if there was a simple way of
> applying these changes we wrote in Java across all the other languages.
Making it easy for clients I think is key... one should be able to
update any node in the solr cluster and have solr ta
Hey guys!
On Thu, 2011-01-27 at 10:04 +1300, Todd Nine wrote:
> Just throwing in my 2 cents. If you're on a tight deadline have you
> had a look at Solandra? We were already using Cassandra, so it was
> incredibly easy to get a scalable Solr installation up and running.
>
In short: We are doin
Just throwing in my 2 cents. If you're on a tight deadline have you had a
look at Solandra? We were already using Cassandra, so it was incredibly
easy to get a scalable Solr installation up and running.
On 27 January 2011 08:17, Alex Cowell wrote:
> Hi Soheb,
>
> Sounds good! A few things I th
Hi Soheb,
Sounds good! A few things I thought of:
With regard to #1, would the list of shards to index to (if present) be
exclusive or would we assume that the shard the update request was sent to
should also be included? For example, say, using the example you gave, an
update request was sent li
31 matches
Mail list logo
