these suggestions would be great to put in a faq!
thanx ted
ben
Ted Dunning wrote:
I always used a large node for ZK to avoid sharing the machine, but the
reason for doing that turned out to be incorrect. In fact, my problem was
to do with GC on the client side.
I can't believe that they are
Ted that's great feedback. I identified a couple of additional things to
verify after reading your comments:
1) ensure that you don't have debug level logging turned on, see this:
https://issues.apache.org/jira/browse/ZOOKEEPER-518
(fixed in 3.2.1, but in general you probably don't want to run a
I always used a large node for ZK to avoid sharing the machine, but the
reason for doing that turned out to be incorrect. In fact, my problem was
to do with GC on the client side.
I can't believe that they are seeing 50 second delays in EC2 due to I/O
contention. GC can do that, but only on a la
How large/small are the writes?
Can you run the following, then try your test again and report the
results for the time period where your test is running?
iostat -x -d 1
also note that ZK JMX allows you to reset the latency attributes (look
under "operations" in jconsole). If you reset the l
Can you enable verboseGC and look at the tenuring distribution and times for
GC?
On Tue, Sep 1, 2009 at 5:54 PM, Satish Bhatti wrote:
> Parallel/Serial.
> inf...@domu-12-31-39-06-3d-d1:/opt/ir/agent/infact-installs/aaa/infact$
> iostat
> Linux 2.6.18-xenU-ec2-v1.0 (domU-12-31-39-06-3D-D1)
Parallel/Serial.
inf...@domu-12-31-39-06-3d-d1:/opt/ir/agent/infact-installs/aaa/infact$
iostat
Linux 2.6.18-xenU-ec2-v1.0 (domU-12-31-39-06-3D-D1) 09/01/2009
_x86_64_
avg-cpu: %user %nice %system %iowait %steal %idle
66.110.001.542.96 20.309.08
Device:
Hi satish,
what GC are you using? Is it ConcurrentMarkSweep or Parallel/Serial?
Also, how is your disk usage on this machine? Can you check your iostat
numbers?
Thanks
mahadev
On 9/1/09 5:15 PM, "Satish Bhatti" wrote:
> GC Time: 11.628 seconds on PS MarkSweep (389 collections)5 minutes on
According to the jconsole, max memory usage is 30MB, 14 live threads and
peak CPU usage about 4%, average is under 1%. We are not really hammering
it. Doing about 10 read/writes every second max.
On Tue, Sep 1, 2009 at 5:20 PM, Ted Dunning wrote:
> This is outrageously large. Max should be mo
This is outrageously large. Max should be more like 50ms.
Either you are doing this somehow, or you have an anomaly on your ZK
machine.
How much data is in ZK? How many transaction per seoncd?
On Tue, Sep 1, 2009 at 5:11 PM, Satish Bhatti wrote:
> MaxRequestLatency 55767
--
Ted Dunning,
I just checked the JMX console.
AvgRequestLatency 38
MaxRequestLatency 55767
I assume those units are milliseconds?
On Tue, Sep 1, 2009 at 5:05 PM, Patrick Hunt wrote:
> Yes. create/set/delete/... are really the issue (non-idempotent).
>
>
> Satish Bhatti wrote:
>
>> Well a bunch of the Connect
GC Time: 11.628 seconds on PS MarkSweep (389 collections)5 minutes on PS
scavenge( 7,636 collections)
It's been running for about 48 hours.
On Tue, Sep 1, 2009 at 5:12 PM, Ted Dunning wrote:
> Do you have long GC delays?
>
> On Tue, Sep 1, 2009 at 4:51 PM, Satish Bhatti wrote:
>
> > Session t
Do you have long GC delays?
On Tue, Sep 1, 2009 at 4:51 PM, Satish Bhatti wrote:
> Session timeout is 30 seconds.
>
> On Tue, Sep 1, 2009 at 4:26 PM, Patrick Hunt wrote:
>
> > What is your client timeout? It may be too low.
> >
> > also see this section on handling recoverable errors:
> > http:
Depends on what your tests are. Are they pretty simple/light? then
probably network issue. Heavy load testing? then might be the
server/client, might be the network.
easiest thing is to run a ping test while running your zk test and see
if pings are getting through (and latency). You should al
Yes. create/set/delete/... are really the issue (non-idempotent).
Satish Bhatti wrote:
Well a bunch of the ConnectionLosses were for zookeeper.exists() calls. I'm
pretty sure dumb retry for those should suffice!
On Tue, Sep 1, 2009 at 4:31 PM, Mahadev Konar wrote:
Hi Satish,
Connectionlos
For my initial testing I am running with a single ZooKeeper server, i.e. the
ensemble only has one server. Not sure if this is exacerbating the problem?
I will check out the trouble shooting link you sent me.
On Tue, Sep 1, 2009 at 5:01 PM, Patrick Hunt wrote:
> I'm not very familiar with ec2
Well a bunch of the ConnectionLosses were for zookeeper.exists() calls. I'm
pretty sure dumb retry for those should suffice!
On Tue, Sep 1, 2009 at 4:31 PM, Mahadev Konar wrote:
> Hi Satish,
>
> Connectionloss is a little trickier than just retrying blindly. Please
> read the following section
I'm not very familiar with ec2 environment, are you doing any
monitoring? In particular network connectivity btw nodes? Sounds like
networking issues btw nodes (I'm assuming you've also looked at stuff
like this http://wiki.apache.org/hadoop/ZooKeeper/Troubleshooting and
verified that you are n
Session timeout is 30 seconds.
On Tue, Sep 1, 2009 at 4:26 PM, Patrick Hunt wrote:
> What is your client timeout? It may be too low.
>
> also see this section on handling recoverable errors:
> http://wiki.apache.org/hadoop/ZooKeeper/ErrorHandling
>
> connection loss in particular needs special c
Hi Satish,
Connectionloss is a little trickier than just retrying blindly. Please
read the following sections on this -
http://wiki.apache.org/hadoop/ZooKeeper/ErrorHandling
And the programmers guide:
http://hadoop.apache.org/zookeeper/docs/r3.1.1/zookeeperProgrammers.html
To learn more abou
What is your client timeout? It may be too low.
also see this section on handling recoverable errors:
http://wiki.apache.org/hadoop/ZooKeeper/ErrorHandling
connection loss in particular needs special care since:
"When a ZooKeeper client loses a connection to the ZooKeeper server
there may be so
I have recently started running on EC2 and am seeing quite a few
ConnectionLoss exceptions. Should I just catch these and retry? Since I
assume that eventually, if the shit truly hits the fan, I will get a
SessionExpired?
Satish
On Mon, Jul 6, 2009 at 11:35 AM, Ted Dunning wrote:
> We have use
Henry Robinson wrote:
Effectively, EC2 does not introduce any new failure modes but potentially
exacerbates some existing ones. If a majority of EC2 nodes fail (in the
sense that their hard drive images cannot be recovered), there is no way to
restart the cluster, and persistence is lost. As you
On Mon, Jul 6, 2009 at 10:16 PM, Ted Dunning wrote:
> No. This should not cause data loss.
> As soon as ZK cannot replicate changes to a majority of machines, it
> refuses
> to take any more changes. This is old ground and is required for
> correctness in the face of network partition. It is
There is no full stop for our system. It (had better) will run forever.
That said, we have no permanent information in ZK that is not persisted in,
say, SVN.
On Mon, Jul 6, 2009 at 3:06 PM, Gustavo Niemeyer wrote:
> > Doing that would make the intricate and unlikely failure mode that Henry
> >
Hi again,
(...)
> ZK seemed pretty darned stable through all of this.
Sounds like a nice test, and it's great to hear that ZooKeeper works well there.
> The only instability that I saw was caused by excessive amounts of data in
> ZK itself. As I neared the (small) amount of memory I had allocat
On Mon, Jul 6, 2009 at 12:58 PM, Gustavo Niemeyer wrote:
> > can make the ZK servers appear a bit less connected. You have to plan
> for
> > ConnectionLoss events.
>
> Interesting.
Note that most of these seem to be related to client issues, especially GC.
If you configure in such a way as to g
No. This should not cause data loss.
As soon as ZK cannot replicate changes to a majority of machines, it refuses
to take any more changes. This is old ground and is required for
correctness in the face of network partition. It is conceivable (barely)
that *exactly* the minority that were behin
Ted thanks for the info.
I've created a wiki page
http://wiki.apache.org/hadoop/ZooKeeper/ZooKeeperOnEC2
to capture details of running ZK on EC2. If you or anyone else would
like to update it with information please do so.
Regards,
Patrick
Ted Dunning wrote:
I disagree with the original pos
On Jul 6, 2009, at 15:40 , Henry Robinson wrote:
This is an interesting way of doing things. It seems like there is a
correctness issue: if a majority of servers fail, with the remaining
minority lagging the leader for some reason, won't the ensemble's
current
state be forever lost? This is aki
Hi Ted,
> b) EC2 interconnect has a lot more going on than in a dedicated VLAN. That
> can make the ZK servers appear a bit less connected. You have to plan for
> ConnectionLoss events.
Interesting.
> c) for highest reliability, I switched to large instances. On reflection, I
> think that was
On Mon, Jul 6, 2009 at 7:38 PM, Ted Dunning wrote:
>
> I think that the misunderstanding is that this on-disk image is critical to
> cluster function. It is not critical because it is replicated to all
> cluster members. This means that any member can disappear and a new
> instance can replace
I disagree with the original post that this is a problem, even in EC2.
Having the persistent copy on disk is exactly what makes the rolling restart
work so well.
I think that the misunderstanding is that this on-disk image is critical to
cluster function. It is not critical because it is replicat
We have used EC2 quite a bit for ZK.
The basic lessons that I have learned include:
a) EC2's biggest advantage after scaling and elasticity was conformity of
configuration. Since you are bringing machines up and down all the time,
they begin to act more like programs and you wind up with boot sc
Hi David,
Answers in line:
On 7/6/09 4:45 AM, "David Graf" wrote:
> Hello
>
> I wanna set up a zookeeper ensemble on amazon's ec2 service. In my
> system, zookeeper is used to run a locking service and to generate
> unique id's. Currently, for testing purposes, I am only running one
> instanc
34 matches
Mail list logo
