Ah, okay - you are setting the shards in solr.xml - thats still an option
to force a node to a particular shard - but if you take that out, shards
will be auto assigned.
By the way, because of the version code, distrib deletes don't work at the
moment - will get to that next week.
- Mark
On Fri, Dec 2, 2011 at 1:16 PM, Jamie Johnson <jej2...@gmail.com> wrote:
> So I'm a fool. I did set the numShards, the issue was so trivial it's
> embarrassing. I did indeed have it setup as a replica, the shard
> names in solr.xml were both shard1. This worked as I expected now.
>
> On Fri, Dec 2, 2011 at 1:02 PM, Mark Miller <markrmil...@gmail.com> wrote:
> >
> > They are unused params, so removing them wouldn't help anything.
> >
> > You might just want to wait till we are further along before playing
> with it.
> >
> > Or if you submit your full self contained test, I can see what's going
> on (eg its still unclear if you have started setting numShards?).
> >
> > I can do a similar set of actions in my tests and it works fine. The
> only reason I could see things working like this is if it thinks you have
> one shard - a leader and a replica.
> >
> > - Mark
> >
> > On Dec 2, 2011, at 12:41 PM, Jamie Johnson wrote:
> >
> >> Glad to hear I don't need to set shards/self, but removing them didn't
> >> seem to change what I'm seeing. Doing this still results in 2
> >> documents 1 on 8983 and 1 on 7574.
> >>
> >> String key = "1";
> >>
> >> SolrInputDocument solrDoc = new SolrInputDocument();
> >> solrDoc.setField("key", key);
> >>
> >> solrDoc.addField("content_mvtxt", "initial value");
> >>
> >> SolrServer server = servers.get("
> http://localhost:8983/solr/collection1";);
> >>
> >> UpdateRequest ureq = new UpdateRequest();
> >> ureq.setParam("update.chain", "distrib-update-chain");
> >> ureq.add(solrDoc);
> >> ureq.setAction(ACTION.COMMIT, true, true);
> >> server.request(ureq);
> >> server.commit();
> >>
> >> solrDoc = new SolrInputDocument();
> >> solrDoc.addField("key", key);
> >> solrDoc.addField("content_mvtxt", "updated value");
> >>
> >> server = servers.get("
> http://localhost:7574/solr/collection1";);
> >>
> >> ureq = new UpdateRequest();
> >> ureq.setParam("update.chain", "distrib-update-chain");
> >> ureq.add(solrDoc);
> >> ureq.setAction(ACTION.COMMIT, true, true);
> >> server.request(ureq);
> >> server.commit();
> >>
> >> server = servers.get("
> http://localhost:8983/solr/collection1";);
> >>
> >>
> >> server.commit();
> >> System.out.println("done");
> >>
> >> On Fri, Dec 2, 2011 at 10:48 AM, Mark Miller <markrmil...@gmail.com>
> wrote:
> >>> So I dunno. You are running a zk server and running in zk mode right?
> >>>
> >>> You don't need to / shouldn't set a shards or self param. The shards
> are
> >>> figured out from Zookeeper.
> >>>
> >>> You always want to use the distrib-update-chain. Eventually it will
> >>> probably be part of the default chain and auto turn in zk mode.
> >>>
> >>> If you are running in zk mode attached to a zk server, this should
> work no
> >>> problem. You can add docs to any server and they will be forwarded to
> the
> >>> correct shard leader and then versioned and forwarded to replicas.
> >>>
> >>> You can also use the CloudSolrServer solrj client - that way you don't
> even
> >>> have to choose a server to send docs too - in which case if it went
> down
> >>> you would have to choose another manually - CloudSolrServer
> automatically
> >>> finds one that is up through ZooKeeper. Eventually it will also be
> smart
> >>> and do the hashing itself so that it can send directly to the shard
> leader
> >>> that the doc would be forwarded to anyway.
> >>>
> >>> - Mark
> >>>
> >>> On Fri, Dec 2, 2011 at 12:09 AM, Jamie Johnson <jej2...@gmail.com>
> wrote:
> >>>
> >>>> Really just trying to do a simple add and update test, the chain
> >>>> missing is just proof of my not understanding exactly how this is
> >>>> supposed to work. I modified the code to this
> >>>>
> >>>> String key = "1";
> >>>>
> >>>> SolrInputDocument solrDoc = new SolrInputDocument();
> >>>> solrDoc.setField("key", key);
> >>>>
> >>>> solrDoc.addField("content_mvtxt", "initial value");
> >>>>
> >>>> SolrServer server = servers
> >>>> .get("
> >>>> http://localhost:8983/solr/collection1";);
> >>>>
> >>>> UpdateRequest ureq = new UpdateRequest();
> >>>> ureq.setParam("update.chain", "distrib-update-chain");
> >>>> ureq.add(solrDoc);
> >>>> ureq.setParam("shards",
> >>>>
> >>>> "localhost:8983/solr/collection1,localhost:7574/solr/collection1");
> >>>> ureq.setParam("self", "foo");
> >>>> ureq.setAction(ACTION.COMMIT, true, true);
> >>>> server.request(ureq);
> >>>> server.commit();
> >>>>
> >>>> solrDoc = new SolrInputDocument();
> >>>> solrDoc.addField("key", key);
> >>>> solrDoc.addField("content_mvtxt", "updated value");
> >>>>
> >>>> server = servers.get("
> >>>> http://localhost:7574/solr/collection1";);
> >>>>
> >>>> ureq = new UpdateRequest();
> >>>> ureq.setParam("update.chain", "distrib-update-chain");
> >>>> //
> ureq.deleteById("8060a9eb-9546-43ee-95bb-d18ea26a6285");
> >>>> ureq.add(solrDoc);
> >>>> ureq.setParam("shards",
> >>>>
> >>>> "localhost:8983/solr/collection1,localhost:7574/solr/collection1");
> >>>> ureq.setParam("self", "foo");
> >>>> ureq.setAction(ACTION.COMMIT, true, true);
> >>>> server.request(ureq);
> >>>> // server.add(solrDoc);
> >>>> server.commit();
> >>>> server = servers.get("
> >>>> http://localhost:8983/solr/collection1";);
> >>>>
> >>>>
> >>>> server.commit();
> >>>> System.out.println("done");
> >>>>
> >>>> but I'm still seeing the doc appear on both shards. After the first
> >>>> commit I see the doc on 8983 with "initial value". after the second
> >>>> commit I see the updated value on 7574 and the old on 8983. After the
> >>>> final commit the doc on 8983 gets updated.
> >>>>
> >>>> Is there something wrong with my test?
> >>>>
> >>>> On Thu, Dec 1, 2011 at 11:17 PM, Mark Miller <markrmil...@gmail.com>
> >>>> wrote:
> >>>>> Getting late - didn't really pay attention to your code I guess - why
> >>>> are you adding the first doc without specifying the distrib update
> chain?
> >>>> This is not really supported. It's going to just go to the server you
> >>>> specified - even with everything setup right, the update might then
> go to
> >>>> that same server or the other one depending on how it hashes. You
> really
> >>>> want to just always use the distrib update chain. I guess I don't yet
> >>>> understand what you are trying to test.
> >>>>>
> >>>>> Sent from my iPad
> >>>>>
> >>>>> On Dec 1, 2011, at 10:57 PM, Mark Miller <markrmil...@gmail.com>
> wrote:
> >>>>>
> >>>>>> Not sure offhand - but things will be funky if you don't specify the
> >>>> correct numShards.
> >>>>>>
> >>>>>> The instance to shard assignment should be using numShards to
> assign.
> >>>> But then the hash to shard mapping actually goes on the number of
> shards it
> >>>> finds registered in ZK (it doesn't have to, but really these should be
> >>>> equal).
> >>>>>>
> >>>>>> So basically you are saying, I want 3 partitions, but you are only
> >>>> starting up 2 nodes, and the code is just not happy about that I'd
> guess.
> >>>> For the system to work properly, you have to fire up at least as many
> >>>> servers as numShards.
> >>>>>>
> >>>>>> What are you trying to do? 2 partitions with no replicas, or one
> >>>> partition with one replica?
> >>>>>>
> >>>>>> In either case, I think you will have better luck if you fire up at
> >>>> least as many servers as the numShards setting. Or lower the numShards
> >>>> setting.
> >>>>>>
> >>>>>> This is all a work in progress by the way - what you are trying to
> test
> >>>> should work if things are setup right though.
> >>>>>>
> >>>>>> - Mark
> >>>>>>
> >>>>>>
> >>>>>> On Dec 1, 2011, at 10:40 PM, Jamie Johnson wrote:
> >>>>>>
> >>>>>>> Thanks for the quick response. With that change (have not done
> >>>>>>> numShards yet) shard1 got updated. But now when executing the
> >>>>>>> following queries I get information back from both, which doesn't
> seem
> >>>>>>> right
> >>>>>>>
> >>>>>>> http://localhost:7574/solr/select/?q=*:*
> >>>>>>> <doc><str name="key">1</str><str name="content_mvtxt">updated
> >>>> value</str></doc>
> >>>>>>>
> >>>>>>> http://localhost:8983/solr/select?q=*:*
> >>>>>>> <doc><str name="key">1</str><str name="content_mvtxt">updated
> >>>> value</str></doc>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Thu, Dec 1, 2011 at 10:21 PM, Mark Miller <
> markrmil...@gmail.com>
> >>>> wrote:
> >>>>>>>> Hmm...sorry bout that - so my first guess is that right now we are
> >>>> not distributing a commit (easy to add, just have not done it).
> >>>>>>>>
> >>>>>>>> Right now I explicitly commit on each server for tests.
> >>>>>>>>
> >>>>>>>> Can you try explicitly committing on server1 after updating the
> doc
> >>>> on server 2?
> >>>>>>>>
> >>>>>>>> I can start distributing commits tomorrow - been meaning to do it
> for
> >>>> my own convenience anyhow.
> >>>>>>>>
> >>>>>>>> Also, you want to pass the sys property numShards=1 on startup. I
> >>>> think it defaults to 3. That will give you one leader and one replica.
> >>>>>>>>
> >>>>>>>> - Mark
> >>>>>>>>
> >>>>>>>> On Dec 1, 2011, at 9:56 PM, Jamie Johnson wrote:
> >>>>>>>>
> >>>>>>>>> So I couldn't resist, I attempted to do this tonight, I used the
> >>>>>>>>> solrconfig you mentioned (as is, no modifications), I setup a 2
> shard
> >>>>>>>>> cluster in collection1, I sent 1 doc to 1 of the shards, updated
> it
> >>>>>>>>> and sent the update to the other. I don't see the modifications
> >>>>>>>>> though I only see the original document. The following is the
> test
> >>>>>>>>>
> >>>>>>>>> public void update() throws Exception {
> >>>>>>>>>
> >>>>>>>>> String key = "1";
> >>>>>>>>>
> >>>>>>>>> SolrInputDocument solrDoc = new SolrInputDocument();
> >>>>>>>>> solrDoc.setField("key", key);
> >>>>>>>>>
> >>>>>>>>> solrDoc.addField("content", "initial value");
> >>>>>>>>>
> >>>>>>>>> SolrServer server = servers
> >>>>>>>>> .get("
> >>>> http://localhost:8983/solr/collection1";);
> >>>>>>>>> server.add(solrDoc);
> >>>>>>>>>
> >>>>>>>>> server.commit();
> >>>>>>>>>
> >>>>>>>>> solrDoc = new SolrInputDocument();
> >>>>>>>>> solrDoc.addField("key", key);
> >>>>>>>>> solrDoc.addField("content", "updated value");
> >>>>>>>>>
> >>>>>>>>> server = servers.get("
> >>>> http://localhost:7574/solr/collection1";);
> >>>>>>>>>
> >>>>>>>>> UpdateRequest ureq = new UpdateRequest();
> >>>>>>>>> ureq.setParam("update.chain",
> "distrib-update-chain");
> >>>>>>>>> ureq.add(solrDoc);
> >>>>>>>>> ureq.setParam("shards",
> >>>>>>>>>
> >>>> "localhost:8983/solr/collection1,localhost:7574/solr/collection1");
> >>>>>>>>> ureq.setParam("self", "foo");
> >>>>>>>>> ureq.setAction(ACTION.COMMIT, true, true);
> >>>>>>>>> server.request(ureq);
> >>>>>>>>> System.out.println("done");
> >>>>>>>>> }
> >>>>>>>>>
> >>>>>>>>> key is my unique field in schema.xml
> >>>>>>>>>
> >>>>>>>>> What am I doing wrong?
> >>>>>>>>>
> >>>>>>>>> On Thu, Dec 1, 2011 at 8:51 PM, Jamie Johnson <jej2...@gmail.com
> >
> >>>> wrote:
> >>>>>>>>>> Yes, the ZK method seems much more flexible. Adding a new shard
> >>>> would
> >>>>>>>>>> be simply updating the range assignments in ZK. Where is this
> >>>>>>>>>> currently on the list of things to accomplish? I don't have
> time to
> >>>>>>>>>> work on this now, but if you (or anyone) could provide
> direction I'd
> >>>>>>>>>> be willing to work on this when I had spare time. I guess a
> JIRA
> >>>>>>>>>> detailing where/how to do this could help. Not sure if the
> design
> >>>> has
> >>>>>>>>>> been thought out that far though.
> >>>>>>>>>>
> >>>>>>>>>> On Thu, Dec 1, 2011 at 8:15 PM, Mark Miller <
> markrmil...@gmail.com>
> >>>> wrote:
> >>>>>>>>>>> Right now lets say you have one shard - everything there
> hashes to
> >>>> range X.
> >>>>>>>>>>>
> >>>>>>>>>>> Now you want to split that shard with an Index Splitter.
> >>>>>>>>>>>
> >>>>>>>>>>> You divide range X in two - giving you two ranges - then you
> start
> >>>> splitting. This is where the current Splitter needs a little
> modification.
> >>>> You decide which doc should go into which new index by rehashing each
> doc
> >>>> id in the index you are splitting - if its hash is greater than X/2,
> it
> >>>> goes into index1 - if its less, index2. I think there are a couple
> current
> >>>> Splitter impls, but one of them does something like, give me an id -
> now if
> >>>> the id's in the index are above that id, goto index1, if below,
> index2. We
> >>>> need to instead do a quick hash rather than simple id compare.
> >>>>>>>>>>>
> >>>>>>>>>>> Why do you need to do this on every shard?
> >>>>>>>>>>>
> >>>>>>>>>>> The other part we need that we dont have is to store hash range
> >>>> assignments in zookeeper - we don't do that yet because it's not
> needed
> >>>> yet. Instead we currently just simply calculate that on the fly (too
> often
> >>>> at the moment - on every request :) I intend to fix that of course).
> >>>>>>>>>>>
> >>>>>>>>>>> At the start, zk would say, for range X, goto this shard. After
> >>>> the split, it would say, for range less than X/2 goto the old node,
> for
> >>>> range greater than X/2 goto the new node.
> >>>>>>>>>>>
> >>>>>>>>>>> - Mark
> >>>>>>>>>>>
> >>>>>>>>>>> On Dec 1, 2011, at 7:44 PM, Jamie Johnson wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> hmmm.....This doesn't sound like the hashing algorithm that's
> on
> >>>> the
> >>>>>>>>>>>> branch, right? The algorithm you're mentioning sounds like
> there
> >>>> is
> >>>>>>>>>>>> some logic which is able to tell that a particular range
> should be
> >>>>>>>>>>>> distributed between 2 shards instead of 1. So seems like a
> trade
> >>>> off
> >>>>>>>>>>>> between repartitioning the entire index (on every shard) and
> >>>> having a
> >>>>>>>>>>>> custom hashing algorithm which is able to handle the situation
> >>>> where 2
> >>>>>>>>>>>> or more shards map to a particular range.
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Thu, Dec 1, 2011 at 7:34 PM, Mark Miller <
> >>>> markrmil...@gmail.com> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Dec 1, 2011, at 7:20 PM, Jamie Johnson wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> I am not familiar with the index splitter that is in
> contrib,
> >>>> but I'll
> >>>>>>>>>>>>>> take a look at it soon. So the process sounds like it
> would be
> >>>> to run
> >>>>>>>>>>>>>> this on all of the current shards indexes based on the hash
> >>>> algorithm.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Not something I've thought deeply about myself yet, but I
> think
> >>>> the idea would be to split as many as you felt you needed to.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> If you wanted to keep the full balance always, this would
> mean
> >>>> splitting every shard at once, yes. But this depends on how many boxes
> >>>> (partitions) you are willing/able to add at a time.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> You might just split one index to start - now it's hash range
> >>>> would be handled by two shards instead of one (if you have 3 replicas
> per
> >>>> shard, this would mean adding 3 more boxes). When you needed to expand
> >>>> again, you would split another index that was still handling its full
> >>>> starting range. As you grow, once you split every original index,
> you'd
> >>>> start again, splitting one of the now half ranges.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Is there also an index merger in contrib which could be
> used to
> >>>> merge
> >>>>>>>>>>>>>> indexes? I'm assuming this would be the process?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> You can merge with IndexWriter.addIndexes (Solr also has an
> >>>> admin command that can do this). But I'm not sure where this fits in?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> - Mark
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Thu, Dec 1, 2011 at 7:18 PM, Mark Miller <
> >>>> markrmil...@gmail.com> wrote:
> >>>>>>>>>>>>>>> Not yet - we don't plan on working on this until a lot of
> >>>> other stuff is
> >>>>>>>>>>>>>>> working solid at this point. But someone else could jump
> in!
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> There are a couple ways to go about it that I know of:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> A more long term solution may be to start using micro
> shards -
> >>>> each index
> >>>>>>>>>>>>>>> starts as multiple indexes. This makes it pretty fast to
> move
> >>>> mirco shards
> >>>>>>>>>>>>>>> around as you decide to change partitions. It's also less
> >>>> flexible as you
> >>>>>>>>>>>>>>> are limited by the number of micro shards you start with.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> A more simple and likely first step is to use an index
> >>>> splitter . We
> >>>>>>>>>>>>>>> already have one in lucene contrib - we would just need to
> >>>> modify it so
> >>>>>>>>>>>>>>> that it splits based on the hash of the document id. This
> is
> >>>> super
> >>>>>>>>>>>>>>> flexible, but splitting will obviously take a little while
> on
> >>>> a huge index.
> >>>>>>>>>>>>>>> The current index splitter is a multi pass splitter - good
> >>>> enough to start
> >>>>>>>>>>>>>>> with, but most files under codec control these days, we
> may be
> >>>> able to make
> >>>>>>>>>>>>>>> a single pass splitter soon as well.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Eventually you could imagine using both options - micro
> shards
> >>>> that could
> >>>>>>>>>>>>>>> also be split as needed. Though I still wonder if micro
> shards
> >>>> will be
> >>>>>>>>>>>>>>> worth the extra complications myself...
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Right now though, the idea is that you should pick a good
> >>>> number of
> >>>>>>>>>>>>>>> partitions to start given your expected data ;) Adding more
> >>>> replicas is
> >>>>>>>>>>>>>>> trivial though.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> - Mark
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Thu, Dec 1, 2011 at 6:35 PM, Jamie Johnson <
> >>>> jej2...@gmail.com> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Another question, is there any support for repartitioning
> of
> >>>> the index
> >>>>>>>>>>>>>>>> if a new shard is added? What is the recommended
> approach for
> >>>>>>>>>>>>>>>> handling this? It seemed that the hashing algorithm (and
> >>>> probably
> >>>>>>>>>>>>>>>> any) would require the index to be repartitioned should a
> new
> >>>> shard be
> >>>>>>>>>>>>>>>> added.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Thu, Dec 1, 2011 at 6:32 PM, Jamie Johnson <
> >>>> jej2...@gmail.com> wrote:
> >>>>>>>>>>>>>>>>> Thanks I will try this first thing in the morning.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Thu, Dec 1, 2011 at 3:39 PM, Mark Miller <
> >>>> markrmil...@gmail.com>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>> On Thu, Dec 1, 2011 at 10:08 AM, Jamie Johnson <
> >>>> jej2...@gmail.com>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> I am currently looking at the latest solrcloud branch
> and
> >>>> was
> >>>>>>>>>>>>>>>>>>> wondering if there was any documentation on
> configuring the
> >>>>>>>>>>>>>>>>>>> DistributedUpdateProcessor? What specifically in
> >>>> solrconfig.xml needs
> >>>>>>>>>>>>>>>>>>> to be added/modified to make distributed indexing work?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Hi Jaime - take a look at solrconfig-distrib-update.xml
> in
> >>>>>>>>>>>>>>>>>> solr/core/src/test-files
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> You need to enable the update log, add an empty
> replication
> >>>> handler def,
> >>>>>>>>>>>>>>>>>> and an update chain with
> >>>> solr.DistributedUpdateProcessFactory in it.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>> - Mark
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> http://www.lucidimagination.com
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>> - Mark
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> http://www.lucidimagination.com
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> - Mark Miller
> >>>>>>>>>>>>> lucidimagination.com
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> - Mark Miller
> >>>>>>>>>>> lucidimagination.com
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>> - Mark Miller
> >>>>>>>> lucidimagination.com
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>> - Mark Miller
> >>>>>> lucidimagination.com
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> - Mark
> >>>
> >>> http://www.lucidimagination.com
> >>>
> >
> > - Mark Miller
> > lucidimagination.com
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
>
--
- Mark
http://www.lucidimagination.com
