Re: Collection will not replicate
I brought down the afflicted servers, waited 5 min, then brought them back up very slowly. That fixed the problem. The bad shard was assigned a leader. Great advice as usual. Erick Erickson wrote > Yes. If indexing went through you'd lose docs so indexing will fail. > Querying will fail too unless you set shards.tolerant. > > You really wouldn't want your docs lost is the reasoning. > > On Feb 2, 2017 6:56 AM, "tedsolr" > tsmith@ > wrote: > >> Can I assume that without a leader the shard will not respond to write >> requests? I can search on the collection. If I can't update docs or add >> any >> new docs then this becomes an emergency. >> >> >> Erick Erickson wrote >> > It's worth a try to take down your entire cluster. Bring one machine >> > back up at a time. There _may_ be something like a 3 minute wait >> > before each of the replicas on that machine come up, the leader >> > election process has a 180 second delay before the replicas on that >> > node take over leadership to wait for the last known good leader to >> > come up. >> > >> > Continue bringing one node up at a time and wait patiently until all >> > the replicas on it are green and until you have a leader for each >> > shard elected. Bringing up the rest of the Solr nodes should be >> > quicker then. >> > >> > Be sure to sequence things so you have known good Solr nodes come up >> > first for the shard that's wonky. By that I mean that the first node >> > you bring up for the leaderless shard should be the one with the best >> > chance of having a totally OK index. >> > >> > >> > Let's claim that the above does bring up a leader for each shard. If >> > you still have a replica that refuses to come up, use the >> > DELETEREPLICA command to remove it. Just for insurance, I'd take the >> > Solr node down after the DELETEREPLICA and remove the entire core >> > directory for the replica that didn't come up. Then restart the node >> > and use the ADDREPLICA collections API command to put it back. >> > >> > If none of that works, you could try hand-editing the state.json file >> > and _make_ one of the shards a leader (I'd do this with the Solr nodes >> > down), but that's not for the faint of heart. >> > >> > Best, >> > Erick >> > >> > On Wed, Feb 1, 2017 at 1:57 PM, Jeff Wartes >> >> > jwartes@ >> >> > wrote: >> >> Sounds similar to a thread last year: >> >> http://lucene.472066.n3.nabble.com/Node-not- >> recovering-leader-elections-not-occuring-tp4287819p4287866.html >> >> >> >> >> >> >> >> On 2/1/17, 7:49 AM, "tedsolr" >> >> > tsmith@ >> >> > wrote: >> >> >> >> I have version 5.2.1. Short of an upgrade, are there any remedies? >> >> >> >> >> >> Erick Erickson wrote >> >> > What version of Solr? since 5.4 there's been a FORCELEADER >> >> colelctions >> >> > API call that might help. >> >> > >> >> > I'd run it with the newly added replicas offline. you only want >> it >> >> to >> >> > have good replicas to choose from. >> >> > >> >> > Best, >> >> > Erick >> >> > >> >> > On Wed, Feb 1, 2017 at 6:48 AM, tedsolr >> >> >> >> > tsmith@ >> >> >> >> > wrote: >> >> >> Update! I did find an error: >> >> >> >> >> >> 2017-02-01 09:23:22.673 ERROR org.apache.solr.common. >> SolrException >> >> >> :org.apache.solr.common.SolrException: Error getting leader >> from >> >> zk for >> >> >> shard shard1 >> >> >> >> >> >> Caused by: org.apache.solr.common.SolrException: Could not get >> >> leader >> >> >> props >> >> >> at >> >> >> >> >> org.apache.solr.cloud.ZkController.getLeaderProps( >> ZkController.java:1040) >> >> >> at >> >> >> >> >> org.apache.solr.cloud.ZkController.getLeaderProps( >> ZkController.java:1004) >> >> >> at >> >> >> >> >> org.apache
Re: Collection will not replicate
Yes. If indexing went through you'd lose docs so indexing will fail. Querying will fail too unless you set shards.tolerant. You really wouldn't want your docs lost is the reasoning. On Feb 2, 2017 6:56 AM, "tedsolr" <tsm...@sciquest.com> wrote: > Can I assume that without a leader the shard will not respond to write > requests? I can search on the collection. If I can't update docs or add any > new docs then this becomes an emergency. > > > Erick Erickson wrote > > It's worth a try to take down your entire cluster. Bring one machine > > back up at a time. There _may_ be something like a 3 minute wait > > before each of the replicas on that machine come up, the leader > > election process has a 180 second delay before the replicas on that > > node take over leadership to wait for the last known good leader to > > come up. > > > > Continue bringing one node up at a time and wait patiently until all > > the replicas on it are green and until you have a leader for each > > shard elected. Bringing up the rest of the Solr nodes should be > > quicker then. > > > > Be sure to sequence things so you have known good Solr nodes come up > > first for the shard that's wonky. By that I mean that the first node > > you bring up for the leaderless shard should be the one with the best > > chance of having a totally OK index. > > > > > > Let's claim that the above does bring up a leader for each shard. If > > you still have a replica that refuses to come up, use the > > DELETEREPLICA command to remove it. Just for insurance, I'd take the > > Solr node down after the DELETEREPLICA and remove the entire core > > directory for the replica that didn't come up. Then restart the node > > and use the ADDREPLICA collections API command to put it back. > > > > If none of that works, you could try hand-editing the state.json file > > and _make_ one of the shards a leader (I'd do this with the Solr nodes > > down), but that's not for the faint of heart. > > > > Best, > > Erick > > > > On Wed, Feb 1, 2017 at 1:57 PM, Jeff Wartes > > > jwartes@ > > > wrote: > >> Sounds similar to a thread last year: > >> http://lucene.472066.n3.nabble.com/Node-not- > recovering-leader-elections-not-occuring-tp4287819p4287866.html > >> > >> > >> > >> On 2/1/17, 7:49 AM, "tedsolr" > > > tsmith@ > > > wrote: > >> > >> I have version 5.2.1. Short of an upgrade, are there any remedies? > >> > >> > >> Erick Erickson wrote > >> > What version of Solr? since 5.4 there's been a FORCELEADER > >> colelctions > >> > API call that might help. > >> > > >> > I'd run it with the newly added replicas offline. you only want it > >> to > >> > have good replicas to choose from. > >> > > >> > Best, > >> > Erick > >> > > >> > On Wed, Feb 1, 2017 at 6:48 AM, tedsolr > >> > >> > tsmith@ > >> > >> > wrote: > >> >> Update! I did find an error: > >> >> > >> >> 2017-02-01 09:23:22.673 ERROR org.apache.solr.common. > SolrException > >> >> :org.apache.solr.common.SolrException: Error getting leader from > >> zk for > >> >> shard shard1 > >> >> > >> >> Caused by: org.apache.solr.common.SolrException: Could not get > >> leader > >> >> props > >> >> at > >> >> > >> org.apache.solr.cloud.ZkController.getLeaderProps( > ZkController.java:1040) > >> >> at > >> >> > >> org.apache.solr.cloud.ZkController.getLeaderProps( > ZkController.java:1004) > >> >> at > >> >> > >> org.apache.solr.cloud.ZkController.getLeader(ZkController.java:960) > >> >> ... 14 more > >> >> Caused by: org.apache.zookeeper.KeeperException$NoNodeException: > >> >> KeeperErrorCode = NoNode for /collections/colname/leaders/shard1 > >> >> at > >> >> > >> org.apache.zookeeper.KeeperException.create(KeeperException.java:111) > >> >> > >> >> When I view the cluster status I see that this shard does not > have > >> a > >> >> leader. > >> >> So it appears I need to force the leader designation to the > >> "active" > >> >> replica. How do I do that? > >> >> > >> >> > >> >> > >> >> -- > >> >> View this message in context: > >> >> > >> http://lucene.472066.n3.nabble.com/Collection-will-not-replicate- > tp4318260p4318265.html > >> >> Sent from the Solr - User mailing list archive at Nabble.com. > >> > >> > >> > >> > >> > >> -- > >> View this message in context: > >> http://lucene.472066.n3.nabble.com/Collection-will-not-replicate- > tp4318260p4318283.html > >> Sent from the Solr - User mailing list archive at Nabble.com. > >> > >> > > > > > > -- > View this message in context: http://lucene.472066.n3. > nabble.com/Collection-will-not-replicate-tp4318260p4318479.html > Sent from the Solr - User mailing list archive at Nabble.com. >
Re: Collection will not replicate
Can I assume that without a leader the shard will not respond to write requests? I can search on the collection. If I can't update docs or add any new docs then this becomes an emergency. Erick Erickson wrote > It's worth a try to take down your entire cluster. Bring one machine > back up at a time. There _may_ be something like a 3 minute wait > before each of the replicas on that machine come up, the leader > election process has a 180 second delay before the replicas on that > node take over leadership to wait for the last known good leader to > come up. > > Continue bringing one node up at a time and wait patiently until all > the replicas on it are green and until you have a leader for each > shard elected. Bringing up the rest of the Solr nodes should be > quicker then. > > Be sure to sequence things so you have known good Solr nodes come up > first for the shard that's wonky. By that I mean that the first node > you bring up for the leaderless shard should be the one with the best > chance of having a totally OK index. > > > Let's claim that the above does bring up a leader for each shard. If > you still have a replica that refuses to come up, use the > DELETEREPLICA command to remove it. Just for insurance, I'd take the > Solr node down after the DELETEREPLICA and remove the entire core > directory for the replica that didn't come up. Then restart the node > and use the ADDREPLICA collections API command to put it back. > > If none of that works, you could try hand-editing the state.json file > and _make_ one of the shards a leader (I'd do this with the Solr nodes > down), but that's not for the faint of heart. > > Best, > Erick > > On Wed, Feb 1, 2017 at 1:57 PM, Jeff Wartes > jwartes@ > wrote: >> Sounds similar to a thread last year: >> http://lucene.472066.n3.nabble.com/Node-not-recovering-leader-elections-not-occuring-tp4287819p4287866.html >> >> >> >> On 2/1/17, 7:49 AM, "tedsolr" > tsmith@ > wrote: >> >> I have version 5.2.1. Short of an upgrade, are there any remedies? >> >> >> Erick Erickson wrote >> > What version of Solr? since 5.4 there's been a FORCELEADER >> colelctions >> > API call that might help. >> > >> > I'd run it with the newly added replicas offline. you only want it >> to >> > have good replicas to choose from. >> > >> > Best, >> > Erick >> > >> > On Wed, Feb 1, 2017 at 6:48 AM, tedsolr >> >> > tsmith@ >> >> > wrote: >> >> Update! I did find an error: >> >> >> >> 2017-02-01 09:23:22.673 ERROR org.apache.solr.common.SolrException >> >> :org.apache.solr.common.SolrException: Error getting leader from >> zk for >> >> shard shard1 >> >> >> >> Caused by: org.apache.solr.common.SolrException: Could not get >> leader >> >> props >> >> at >> >> >> org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1040) >> >> at >> >> >> org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1004) >> >> at >> >> >> org.apache.solr.cloud.ZkController.getLeader(ZkController.java:960) >> >> ... 14 more >> >> Caused by: org.apache.zookeeper.KeeperException$NoNodeException: >> >> KeeperErrorCode = NoNode for /collections/colname/leaders/shard1 >> >> at >> >> >> org.apache.zookeeper.KeeperException.create(KeeperException.java:111) >> >> >> >> When I view the cluster status I see that this shard does not have >> a >> >> leader. >> >> So it appears I need to force the leader designation to the >> "active" >> >> replica. How do I do that? >> >> >> >> >> >> >> >> -- >> >> View this message in context: >> >> >> http://lucene.472066.n3.nabble.com/Collection-will-not-replicate-tp4318260p4318265.html >> >> Sent from the Solr - User mailing list archive at Nabble.com. >> >> >> >> >> >> -- >> View this message in context: >> http://lucene.472066.n3.nabble.com/Collection-will-not-replicate-tp4318260p4318283.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> >> -- View this message in context: http://lucene.472066.n3.nabble.com/Collection-will-not-replicate-tp4318260p4318479.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Collection will not replicate
It's worth a try to take down your entire cluster. Bring one machine back up at a time. There _may_ be something like a 3 minute wait before each of the replicas on that machine come up, the leader election process has a 180 second delay before the replicas on that node take over leadership to wait for the last known good leader to come up. Continue bringing one node up at a time and wait patiently until all the replicas on it are green and until you have a leader for each shard elected. Bringing up the rest of the Solr nodes should be quicker then. Be sure to sequence things so you have known good Solr nodes come up first for the shard that's wonky. By that I mean that the first node you bring up for the leaderless shard should be the one with the best chance of having a totally OK index. Let's claim that the above does bring up a leader for each shard. If you still have a replica that refuses to come up, use the DELETEREPLICA command to remove it. Just for insurance, I'd take the Solr node down after the DELETEREPLICA and remove the entire core directory for the replica that didn't come up. Then restart the node and use the ADDREPLICA collections API command to put it back. If none of that works, you could try hand-editing the state.json file and _make_ one of the shards a leader (I'd do this with the Solr nodes down), but that's not for the faint of heart. Best, Erick On Wed, Feb 1, 2017 at 1:57 PM, Jeff Wartes <jwar...@whitepages.com> wrote: > Sounds similar to a thread last year: > http://lucene.472066.n3.nabble.com/Node-not-recovering-leader-elections-not-occuring-tp4287819p4287866.html > > > > On 2/1/17, 7:49 AM, "tedsolr" <tsm...@sciquest.com> wrote: > > I have version 5.2.1. Short of an upgrade, are there any remedies? > > > Erick Erickson wrote > > What version of Solr? since 5.4 there's been a FORCELEADER colelctions > > API call that might help. > > > > I'd run it with the newly added replicas offline. you only want it to > > have good replicas to choose from. > > > > Best, > > Erick > > > > On Wed, Feb 1, 2017 at 6:48 AM, tedsolr > > > tsmith@ > > > wrote: > >> Update! I did find an error: > >> > >> 2017-02-01 09:23:22.673 ERROR org.apache.solr.common.SolrException > >> :org.apache.solr.common.SolrException: Error getting leader from zk for > >> shard shard1 > >> > >> Caused by: org.apache.solr.common.SolrException: Could not get leader > >> props > >> at > >> > org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1040) > >> at > >> > org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1004) > >> at > >> org.apache.solr.cloud.ZkController.getLeader(ZkController.java:960) > >> ... 14 more > >> Caused by: org.apache.zookeeper.KeeperException$NoNodeException: > >> KeeperErrorCode = NoNode for /collections/colname/leaders/shard1 > >> at > >> org.apache.zookeeper.KeeperException.create(KeeperException.java:111) > >> > >> When I view the cluster status I see that this shard does not have a > >> leader. > >> So it appears I need to force the leader designation to the "active" > >> replica. How do I do that? > >> > >> > >> > >> -- > >> View this message in context: > >> > http://lucene.472066.n3.nabble.com/Collection-will-not-replicate-tp4318260p4318265.html > >> Sent from the Solr - User mailing list archive at Nabble.com. > > > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Collection-will-not-replicate-tp4318260p4318283.html > Sent from the Solr - User mailing list archive at Nabble.com. > >
Re: Collection will not replicate
Sounds similar to a thread last year: http://lucene.472066.n3.nabble.com/Node-not-recovering-leader-elections-not-occuring-tp4287819p4287866.html On 2/1/17, 7:49 AM, "tedsolr" <tsm...@sciquest.com> wrote: I have version 5.2.1. Short of an upgrade, are there any remedies? Erick Erickson wrote > What version of Solr? since 5.4 there's been a FORCELEADER colelctions > API call that might help. > > I'd run it with the newly added replicas offline. you only want it to > have good replicas to choose from. > > Best, > Erick > > On Wed, Feb 1, 2017 at 6:48 AM, tedsolr > tsmith@ > wrote: >> Update! I did find an error: >> >> 2017-02-01 09:23:22.673 ERROR org.apache.solr.common.SolrException >> :org.apache.solr.common.SolrException: Error getting leader from zk for >> shard shard1 >> >> Caused by: org.apache.solr.common.SolrException: Could not get leader >> props >> at >> org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1040) >> at >> org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1004) >> at >> org.apache.solr.cloud.ZkController.getLeader(ZkController.java:960) >> ... 14 more >> Caused by: org.apache.zookeeper.KeeperException$NoNodeException: >> KeeperErrorCode = NoNode for /collections/colname/leaders/shard1 >> at >> org.apache.zookeeper.KeeperException.create(KeeperException.java:111) >> >> When I view the cluster status I see that this shard does not have a >> leader. >> So it appears I need to force the leader designation to the "active" >> replica. How do I do that? >> >> >> >> -- >> View this message in context: >> http://lucene.472066.n3.nabble.com/Collection-will-not-replicate-tp4318260p4318265.html >> Sent from the Solr - User mailing list archive at Nabble.com. -- View this message in context: http://lucene.472066.n3.nabble.com/Collection-will-not-replicate-tp4318260p4318283.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: Collection will not replicate
Erick, Thank you for your quick response, I appreciate your help I am new to Solr and not from Java background I have developed the code in Dev on single core and it works perfectly good On QA box I have installed Solr 6.3 on 2 nodes(SolrCloud) Made few changes to the code to fit SolrCloud I have very less time to complete this job in QA Ultimately I need to load these files on realtime, I am thinking of using flume/Kafka for that Just to show the progress I need to complete loading these emails text files using dataimporthandler Herewith I am attaching my code, Please suggest me what could be the issue Regards, ~Sri -Original Message- From: Erick Erickson [mailto:erickerick...@gmail.com] Sent: Wednesday, February 01, 2017 10:39 AM To: solr-user <solr-user@lucene.apache.org> Subject: Re: Collection will not replicate What version of Solr? since 5.4 there's been a FORCELEADER colelctions API call that might help. I'd run it with the newly added replicas offline. you only want it to have good replicas to choose from. Best, Erick On Wed, Feb 1, 2017 at 6:48 AM, tedsolr <tsm...@sciquest.com> wrote: > Update! I did find an error: > > 2017-02-01 09:23:22.673 ERROR org.apache.solr.common.SolrException > :org.apache.solr.common.SolrException: Error getting leader from zk > for shard shard1 > Caused by: org.apache.solr.common.SolrException: Could not get leader props > at > org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1040) > at > org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1004) > at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:960) > ... 14 more > Caused by: org.apache.zookeeper.KeeperException$NoNodeException: > KeeperErrorCode = NoNode for /collections/colname/leaders/shard1 > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:111) > > When I view the cluster status I see that this shard does not have a leader. > So it appears I need to force the leader designation to the "active" > replica. How do I do that? > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Collection-will-not-replicate-tp431 > 8260p4318265.html Sent from the Solr - User mailing list archive at > Nabble.com. application/xml image/svg+xml text/xml application/x-google-gadget application/excel application/xls application/msworddoc application/msworddot application/powerpoint application/ppt application/x-tika-msoffice application/msword application/vnd.ms-excel application/vnd.ms-excel.sheet.binary.macroenabled.12 application/vnd.ms-powerpoint application/vnd.visio application/vnd.ms-outlook application/x-tika-ooxml application/vnd.openxmlformats-package.core-properties+xml application/vnd.openxmlformats-officedocument.spreadsheetml.sheet application/vnd.openxmlformats-officedocument.spreadsheetml.template application/vnd.ms-excel.sheet.macroenabled.12 application/vnd.ms-excel.template.macroenabled.12 application/vnd.ms-excel.addin.macroenabled.12 application/vnd.openxmlformats-officedocument.presentationml.presentation application/vnd.openxmlformats-officedocument.presentationml.template application/vnd.openxmlformats-officedocument.presentationml.slideshow application/vnd.ms-powerpoint.presentation.macroenabled.12 application/vnd.ms-powerpoint.slideshow.macroenabled.12 application/vnd.ms-powerpoint.addin.macroenabled.12 application/vnd.openxmlformats-officedocument.wordprocessingml.document application/vnd.openxmlformats-officedocument.wordprocessingml.template application/vnd.ms-word.document.macroenabled.12 application/vnd.ms-word.template.macroenabled.12 text/html application/rtf application/pdf text/plain script/groovy application/x-groovy application/x-javascript application/javascript text/javascript application/vnd.oasis.opendocument.database application/vnd.sun.xml.writer application/vnd.oasis.opendocument.text application/vnd.oasis.opendocument.graphics application/vnd.oasis.opendocument.presentation application/vnd.oasis.opendocument.spreadsheet application/vnd.oasis.opendocument.chart application/vnd.oasis.opendocument.image application/vnd.oasis.opendocument.formula application/vnd.oasis.opendocument.text-master application/vnd.oasis.opendocument.text-web application/vnd.oasis.opendocument.text-template application/vnd.oasis.opendocument.graphics-template application/vnd.oasis.opendocu
Re: Collection will not replicate
What version of Solr? since 5.4 there's been a FORCELEADER colelctions API call that might help. I'd run it with the newly added replicas offline. you only want it to have good replicas to choose from. Best, Erick On Wed, Feb 1, 2017 at 6:48 AM, tedsolr <tsm...@sciquest.com> wrote: > Update! I did find an error: > > 2017-02-01 09:23:22.673 ERROR org.apache.solr.common.SolrException > :org.apache.solr.common.SolrException: Error getting leader from zk for > shard shard1 > > Caused by: org.apache.solr.common.SolrException: Could not get leader props > at > org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1040) > at > org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1004) > at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:960) > ... 14 more > Caused by: org.apache.zookeeper.KeeperException$NoNodeException: > KeeperErrorCode = NoNode for /collections/colname/leaders/shard1 > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:111) > > When I view the cluster status I see that this shard does not have a leader. > So it appears I need to force the leader designation to the "active" > replica. How do I do that? > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Collection-will-not-replicate-tp4318260p4318265.html > Sent from the Solr - User mailing list archive at Nabble.com.
Re: Collection will not replicate
Update! I did find an error: 2017-02-01 09:23:22.673 ERROR org.apache.solr.common.SolrException :org.apache.solr.common.SolrException: Error getting leader from zk for shard shard1 Caused by: org.apache.solr.common.SolrException: Could not get leader props at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1040) at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1004) at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:960) ... 14 more Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/colname/leaders/shard1 at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) When I view the cluster status I see that this shard does not have a leader. So it appears I need to force the leader designation to the "active" replica. How do I do that? -- View this message in context: http://lucene.472066.n3.nabble.com/Collection-will-not-replicate-tp4318260p4318265.html Sent from the Solr - User mailing list archive at Nabble.com.
Collection will not replicate
I have a collection (1 shard, 2 replicas) that was doing a batch update when one solr host ran out of disk space. The batch job failed at that point, and one replica got corrupted. I deleted the bad replica. I've tried several times since then to add a new replica. The status of the request is "running" for about 30 minutes or so, then it completes but the new replica is always "down" and has 0 documents. The collection only has 15 million docs. Adding a replica to a shard that small should only take a couple minutes. I haven't seen any errors in the solr logs during the replication process. Has anyone seen this behavior before? What should I be looking at for diagnostic purposes? Thanks for the support v5.2.1 -- View this message in context: http://lucene.472066.n3.nabble.com/Collection-will-not-replicate-tp4318260.html Sent from the Solr - User mailing list archive at Nabble.com.
