Re: 答复: flushing + compactions after config change
bq. On Thu, Jun 27, 2013 at 4:27 PM, Viral Bajaria viral.baja...@gmail.com wrote: It's not random, it picks the region with the most data in its memstores. That's weird, because I see some of my regions which receive the least amount of data in a given time period flushing before the regions that are receiving data continuously. I agree with Viral here. When max logs are reached, we look at the oldest wal and see which regions should be flushed in order to get that first wal (read oldest) archived. In your case Viral, these regions could be those which are not receiving many edits when 32 logs have been rolled. It may be very specific to your use case, but you could try playing with max number of logs? May be make them 16, 40, etc? On Fri, Jun 28, 2013 at 4:53 PM, Jean-Daniel Cryans jdcry...@apache.orgwrote: On Fri, Jun 28, 2013 at 2:39 PM, Viral Bajaria viral.baja...@gmail.com wrote: On Fri, Jun 28, 2013 at 9:31 AM, Jean-Daniel Cryans jdcry...@apache.org wrote: On Thu, Jun 27, 2013 at 4:27 PM, Viral Bajaria viral.baja...@gmail.com wrote: It's not random, it picks the region with the most data in its memstores. That's weird, because I see some of my regions which receive the least amount of data in a given time period flushing before the regions that are receiving data continuously. The reason I know this is because of the write pattern. Some of my tables are in catch-up mode i.e. I am ingesting data from the past and they always have something to do. While some tables are not in catch-up mode and are just sitting idle for most of the time. Yet I see high number of flushes for those regions too. I doubt that it's the fact that it's a major compaction that it's making everything worse. When a minor gets promoted into a major it's because we're already going to compact all the files, so we might as well get rid of some deletes at the same time. They are all getting selected because the files are within the selection ratio. I would not focus on this to resolve your problem. I meant worse for my writes not for HBase as a whole. I haven't been closely following this thread, but have you posted a log snippet somewhere? It's usually much more telling and we eliminate a few levels of interpretation. Make sure it's at DEBUG, and that you grab a few hours of activity. Get the GC log for the same time as well. Drop this on a web server or pastebin if it fits. The only log snippet that I posted was the flushing action. Also that log was not everything, I had grep'd a few lines out. Let me collect some more stats here and post it again. I just enabled GC logging on this server, deployed the wrong config out initially which had no GC logging. I am not sure how GC logs will help here given that I am at less than 50% heap space used and so I would doubt a stop the world GC happening. Are you trying to look for some other information ? Just trying to cover all the bases. J-D
Re: 答复: flushing + compactions after config change
Viral, Basically when you increase the memstore flush size ( well ur aim there is to reduce flushes and make data sit in memory for longer time) you need to carefully consider the 2 things 1. What is the max heap and what is the % memory you have allocated max for all the memstores in a RS. And see how many regions and CFs and whether the allocated max memstore memory is enough. Basically you should not make premature flushes because of global heap preassure. As you said this will create many small HFiles and so compactions 2. Check on the max Hlogs setting. Becuase of too many hlogs also there can be a forced flush which also can create small HFiles. The default value is 32. The HLOg rolling will happen when one HLog's size reaches 95% size of ur HDFS block size. (This 95% is also configurable) If you have increased the flush size and heap allocated for memstores, still you can get premature flushes because of this HLog stuff. Also from 94 onwards there is compression feature for HLog.. Have a look at that also. -Anoop- On Fri, Jun 28, 2013 at 6:52 AM, Viral Bajaria viral.baja...@gmail.comwrote: Thanks for the detailed JVM OPTS. On a side note, I doubt the issue with too many hlogs has anything to do with that right ? Because I have enough spare heap space and my Memstore's are not at the lowerLimit of the total available heap (i.e. 0.35) -Viral On Thu, Jun 27, 2013 at 6:09 PM, Azuryy Yu azury...@gmail.com wrote: Hope that helpful.
Re: 答复: flushing + compactions after config change
On Thu, Jun 27, 2013 at 4:27 PM, Viral Bajaria viral.baja...@gmail.com wrote: Hey JD, Thanks for the clarification. I also came across a previous thread which sort of talks about a similar problem. http://mail-archives.apache.org/mod_mbox/hbase-user/201204.mbox/%3ccagptdnfwnrsnqv7n3wgje-ichzpx-cxn1tbchgwrpohgcos...@mail.gmail.com%3E I guess my problem is also similar to the fact that my writes are well distributed and at a given time I could be writing to a lot of regions. Some of the regions receive very little data but since the flush algorithm choose at random what to flush when too many hlogs is hit, it will flush It's not random, it picks the region with the most data in its memstores. a region with less than 10mb of data causing too many small files. This in-turn causes compaction storms where even though major compactions is disabled, some of the minor get upgraded to major and that's when things start getting worse. I doubt that it's the fact that it's a major compaction that it's making everything worse. When a minor gets promoted into a major it's because we're already going to compact all the files, so we might as well get rid of some deletes at the same time. They are all getting selected because the files are within the selection ratio. I would not focus on this to resolve your problem. My compaction queues are still the same and so I doubt I will be coming out of this storm without bumping up max hlogs for now. Reducing regions per server is one option but then I will be wasting my resources since the servers at current load are at 30% CPU and 25% RAM. Maybe I can bump up heap space and give more memory to the the memstore. Sorry, I am just thinking out loud. I haven't been closely following this thread, but have you posted a log snippet somewhere? It's usually much more telling and we eliminate a few levels of interpretation. Make sure it's at DEBUG, and that you grab a few hours of activity. Get the GC log for the same time as well. Drop this on a web server or pastebin if it fits. Thx, J-D
Re: 答复: flushing + compactions after config change
On Fri, Jun 28, 2013 at 9:31 AM, Jean-Daniel Cryans jdcry...@apache.orgwrote: On Thu, Jun 27, 2013 at 4:27 PM, Viral Bajaria viral.baja...@gmail.com wrote: It's not random, it picks the region with the most data in its memstores. That's weird, because I see some of my regions which receive the least amount of data in a given time period flushing before the regions that are receiving data continuously. The reason I know this is because of the write pattern. Some of my tables are in catch-up mode i.e. I am ingesting data from the past and they always have something to do. While some tables are not in catch-up mode and are just sitting idle for most of the time. Yet I see high number of flushes for those regions too. I doubt that it's the fact that it's a major compaction that it's making everything worse. When a minor gets promoted into a major it's because we're already going to compact all the files, so we might as well get rid of some deletes at the same time. They are all getting selected because the files are within the selection ratio. I would not focus on this to resolve your problem. I meant worse for my writes not for HBase as a whole. I haven't been closely following this thread, but have you posted a log snippet somewhere? It's usually much more telling and we eliminate a few levels of interpretation. Make sure it's at DEBUG, and that you grab a few hours of activity. Get the GC log for the same time as well. Drop this on a web server or pastebin if it fits. The only log snippet that I posted was the flushing action. Also that log was not everything, I had grep'd a few lines out. Let me collect some more stats here and post it again. I just enabled GC logging on this server, deployed the wrong config out initially which had no GC logging. I am not sure how GC logs will help here given that I am at less than 50% heap space used and so I would doubt a stop the world GC happening. Are you trying to look for some other information ? Thx, J-D Thanks to everyone on this list for their time! -Viral
Re: 答复: flushing + compactions after config change
On Fri, Jun 28, 2013 at 2:39 PM, Viral Bajaria viral.baja...@gmail.com wrote: On Fri, Jun 28, 2013 at 9:31 AM, Jean-Daniel Cryans jdcry...@apache.orgwrote: On Thu, Jun 27, 2013 at 4:27 PM, Viral Bajaria viral.baja...@gmail.com wrote: It's not random, it picks the region with the most data in its memstores. That's weird, because I see some of my regions which receive the least amount of data in a given time period flushing before the regions that are receiving data continuously. The reason I know this is because of the write pattern. Some of my tables are in catch-up mode i.e. I am ingesting data from the past and they always have something to do. While some tables are not in catch-up mode and are just sitting idle for most of the time. Yet I see high number of flushes for those regions too. I doubt that it's the fact that it's a major compaction that it's making everything worse. When a minor gets promoted into a major it's because we're already going to compact all the files, so we might as well get rid of some deletes at the same time. They are all getting selected because the files are within the selection ratio. I would not focus on this to resolve your problem. I meant worse for my writes not for HBase as a whole. I haven't been closely following this thread, but have you posted a log snippet somewhere? It's usually much more telling and we eliminate a few levels of interpretation. Make sure it's at DEBUG, and that you grab a few hours of activity. Get the GC log for the same time as well. Drop this on a web server or pastebin if it fits. The only log snippet that I posted was the flushing action. Also that log was not everything, I had grep'd a few lines out. Let me collect some more stats here and post it again. I just enabled GC logging on this server, deployed the wrong config out initially which had no GC logging. I am not sure how GC logs will help here given that I am at less than 50% heap space used and so I would doubt a stop the world GC happening. Are you trying to look for some other information ? Just trying to cover all the bases. J-D
答复: flushing + compactions after config change
If reached memstore global up-limit, you'll find Blocking updates on in your files(see MemStoreFlusher.reclaimMemStoreMemory); If it's caused by too many log files, you'll find Too many hlogs: logs=(see HLog.cleanOldLogs) Hope it's helpful for you:) Best, Liang 发件人: Viral Bajaria [viral.baja...@gmail.com] 发送时间: 2013年6月27日 16:18 收件人: user@hbase.apache.org 主题: Re: flushing + compactions after config change Thanks for the quick response Anoop. The current memstore reserved (IIRC) would be 0.35 of total heap right ? The RS total heap is 10231MB, used is at 5000MB. Total number of regions is 217 and there are approx 150 regions with 2 families, ~60 with 1 family and remaining with 3 families. How to check if the flushes are due to too many WAL files ? Does it get logged ? Thanks, Viral On Thu, Jun 27, 2013 at 12:51 AM, Anoop John anoop.hb...@gmail.com wrote: You mean there is enough memstore reserved heap in the RS, so that there wont be premature flushes because of global heap pressure? What is the RS max mem and how many regions and CFs in each? Can you check whether the flushes happening because of too many WAL files? -Anoop-
Re: 答复: flushing + compactions after config change
Thanks Liang! Found the logs. I had gone overboard with my grep's and missed the Too many hlogs line for the regions that I was trying to debug. A few sample log lines: 2013-06-27 07:42:49,602 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Too many hlogs: logs=33, maxlogs=32; forcing flush of 2 regions(s): 0e940167482d42f1999b29a023c7c18a, 3f486a879418257f053aa75ba5b69b14 2013-06-27 08:10:29,996 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Too many hlogs: logs=33, maxlogs=32; forcing flush of 1 regions(s): 0e940167482d42f1999b29a023c7c18a 2013-06-27 08:17:44,719 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Too many hlogs: logs=33, maxlogs=32; forcing flush of 2 regions(s): 0e940167482d42f1999b29a023c7c18a, e380fd8a7174d34feb903baa97564e08 2013-06-27 08:23:45,357 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Too many hlogs: logs=33, maxlogs=32; forcing flush of 3 regions(s): 0e940167482d42f1999b29a023c7c18a, 3f486a879418257f053aa75ba5b69b14, e380fd8a7174d34feb903baa97564e08 Any pointers on what's the best practice for avoiding this scenario ? Thanks, Viral On Thu, Jun 27, 2013 at 1:21 AM, 谢良 xieli...@xiaomi.com wrote: If reached memstore global up-limit, you'll find Blocking updates on in your files(see MemStoreFlusher.reclaimMemStoreMemory); If it's caused by too many log files, you'll find Too many hlogs: logs=(see HLog.cleanOldLogs) Hope it's helpful for you:) Best, Liang
Re: 答复: flushing + compactions after config change
The config hbase.regionserver.maxlogs specifies what is the max #logs and defaults to 32. But remember if there are so many log files to replay then the MTTR will become more (RS down case ) -Anoop- On Thu, Jun 27, 2013 at 1:59 PM, Viral Bajaria viral.baja...@gmail.comwrote: Thanks Liang! Found the logs. I had gone overboard with my grep's and missed the Too many hlogs line for the regions that I was trying to debug. A few sample log lines: 2013-06-27 07:42:49,602 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Too many hlogs: logs=33, maxlogs=32; forcing flush of 2 regions(s): 0e940167482d42f1999b29a023c7c18a, 3f486a879418257f053aa75ba5b69b14 2013-06-27 08:10:29,996 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Too many hlogs: logs=33, maxlogs=32; forcing flush of 1 regions(s): 0e940167482d42f1999b29a023c7c18a 2013-06-27 08:17:44,719 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Too many hlogs: logs=33, maxlogs=32; forcing flush of 2 regions(s): 0e940167482d42f1999b29a023c7c18a, e380fd8a7174d34feb903baa97564e08 2013-06-27 08:23:45,357 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Too many hlogs: logs=33, maxlogs=32; forcing flush of 3 regions(s): 0e940167482d42f1999b29a023c7c18a, 3f486a879418257f053aa75ba5b69b14, e380fd8a7174d34feb903baa97564e08 Any pointers on what's the best practice for avoiding this scenario ? Thanks, Viral On Thu, Jun 27, 2013 at 1:21 AM, 谢良 xieli...@xiaomi.com wrote: If reached memstore global up-limit, you'll find Blocking updates on in your files(see MemStoreFlusher.reclaimMemStoreMemory); If it's caused by too many log files, you'll find Too many hlogs: logs=(see HLog.cleanOldLogs) Hope it's helpful for you:) Best, Liang
Re: 答复: flushing + compactions after config change
hey Viral, Which hbase version are you using? On Thu, Jun 27, 2013 at 5:03 PM, Anoop John anoop.hb...@gmail.com wrote: The config hbase.regionserver.maxlogs specifies what is the max #logs and defaults to 32. But remember if there are so many log files to replay then the MTTR will become more (RS down case ) -Anoop- On Thu, Jun 27, 2013 at 1:59 PM, Viral Bajaria viral.baja...@gmail.com wrote: Thanks Liang! Found the logs. I had gone overboard with my grep's and missed the Too many hlogs line for the regions that I was trying to debug. A few sample log lines: 2013-06-27 07:42:49,602 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Too many hlogs: logs=33, maxlogs=32; forcing flush of 2 regions(s): 0e940167482d42f1999b29a023c7c18a, 3f486a879418257f053aa75ba5b69b14 2013-06-27 08:10:29,996 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Too many hlogs: logs=33, maxlogs=32; forcing flush of 1 regions(s): 0e940167482d42f1999b29a023c7c18a 2013-06-27 08:17:44,719 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Too many hlogs: logs=33, maxlogs=32; forcing flush of 2 regions(s): 0e940167482d42f1999b29a023c7c18a, e380fd8a7174d34feb903baa97564e08 2013-06-27 08:23:45,357 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Too many hlogs: logs=33, maxlogs=32; forcing flush of 3 regions(s): 0e940167482d42f1999b29a023c7c18a, 3f486a879418257f053aa75ba5b69b14, e380fd8a7174d34feb903baa97564e08 Any pointers on what's the best practice for avoiding this scenario ? Thanks, Viral On Thu, Jun 27, 2013 at 1:21 AM, 谢良 xieli...@xiaomi.com wrote: If reached memstore global up-limit, you'll find Blocking updates on in your files(see MemStoreFlusher.reclaimMemStoreMemory); If it's caused by too many log files, you'll find Too many hlogs: logs=(see HLog.cleanOldLogs) Hope it's helpful for you:) Best, Liang
Re: 答复: flushing + compactions after config change
0.94.4 with plans to upgrade to the latest 0.94 release. On Thu, Jun 27, 2013 at 2:22 AM, Azuryy Yu azury...@gmail.com wrote: hey Viral, Which hbase version are you using?
Re: 答复: flushing + compactions after config change
Can you paste your JVM options here? and Do you have an extensive write on your hbase cluster? On Thu, Jun 27, 2013 at 5:47 PM, Viral Bajaria viral.baja...@gmail.comwrote: 0.94.4 with plans to upgrade to the latest 0.94 release. On Thu, Jun 27, 2013 at 2:22 AM, Azuryy Yu azury...@gmail.com wrote: hey Viral, Which hbase version are you using?
Re: 答复: flushing + compactions after config change
I do have a heavy write operation going on. Actually heavy is relative. Not all tables/regions are seeing the same amount of writes at the same time. There is definitely a burst of writes that can happen on some regions. In addition to that there are some processing jobs which play catch up and could be processing data in the past and they could have more heavy write operations. I think my main problem is, my writes are well distributed across regions. A batch of puts most probably end up hitting every region since they get distributed fairly well. In that scenario, I am guessing I get a lot of WALs though I am just speculating. Regarding the JVM options (minus some settings for remote profiling): -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=1 -XX:GCLogFileSize=512M On Thu, Jun 27, 2013 at 2:48 AM, Azuryy Yu azury...@gmail.com wrote: Can you paste your JVM options here? and Do you have an extensive write on your hbase cluster?
答复: 答复: flushing + compactions after config change
btw, don't use CMSIncrementalMode, iirc, it had been removed from hotspot upstream accually. 发件人: Viral Bajaria [viral.baja...@gmail.com] 发送时间: 2013年6月27日 18:08 收件人: user@hbase.apache.org 主题: Re: 答复: flushing + compactions after config change I do have a heavy write operation going on. Actually heavy is relative. Not all tables/regions are seeing the same amount of writes at the same time. There is definitely a burst of writes that can happen on some regions. In addition to that there are some processing jobs which play catch up and could be processing data in the past and they could have more heavy write operations. I think my main problem is, my writes are well distributed across regions. A batch of puts most probably end up hitting every region since they get distributed fairly well. In that scenario, I am guessing I get a lot of WALs though I am just speculating. Regarding the JVM options (minus some settings for remote profiling): -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=1 -XX:GCLogFileSize=512M On Thu, Jun 27, 2013 at 2:48 AM, Azuryy Yu azury...@gmail.com wrote: Can you paste your JVM options here? and Do you have an extensive write on your hbase cluster?
Re: 答复: flushing + compactions after config change
your JVM options arenot enough. I will give you some detail when I go back office tomorrow. --Send from my Sony mobile. On Jun 27, 2013 6:09 PM, Viral Bajaria viral.baja...@gmail.com wrote: I do have a heavy write operation going on. Actually heavy is relative. Not all tables/regions are seeing the same amount of writes at the same time. There is definitely a burst of writes that can happen on some regions. In addition to that there are some processing jobs which play catch up and could be processing data in the past and they could have more heavy write operations. I think my main problem is, my writes are well distributed across regions. A batch of puts most probably end up hitting every region since they get distributed fairly well. In that scenario, I am guessing I get a lot of WALs though I am just speculating. Regarding the JVM options (minus some settings for remote profiling): -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=1 -XX:GCLogFileSize=512M On Thu, Jun 27, 2013 at 2:48 AM, Azuryy Yu azury...@gmail.com wrote: Can you paste your JVM options here? and Do you have an extensive write on your hbase cluster?
Re: 答复: flushing + compactions after config change
Thanks Azuryy. Look forward to it. Does DEFERRED_LOG_FLUSH impact the number of WAL files that will be created ? Tried looking around but could not find the details. On Thu, Jun 27, 2013 at 7:53 AM, Azuryy Yu azury...@gmail.com wrote: your JVM options arenot enough. I will give you some detail when I go back office tomorrow. --Send from my Sony mobile.
Re: 答复: flushing + compactions after config change
No, all your data eventually makes it into the log, just potentially not as quickly :) J-D On Thu, Jun 27, 2013 at 2:06 PM, Viral Bajaria viral.baja...@gmail.com wrote: Thanks Azuryy. Look forward to it. Does DEFERRED_LOG_FLUSH impact the number of WAL files that will be created ? Tried looking around but could not find the details. On Thu, Jun 27, 2013 at 7:53 AM, Azuryy Yu azury...@gmail.com wrote: your JVM options arenot enough. I will give you some detail when I go back office tomorrow. --Send from my Sony mobile.
Re: 答复: flushing + compactions after config change
Hey JD, Thanks for the clarification. I also came across a previous thread which sort of talks about a similar problem. http://mail-archives.apache.org/mod_mbox/hbase-user/201204.mbox/%3ccagptdnfwnrsnqv7n3wgje-ichzpx-cxn1tbchgwrpohgcos...@mail.gmail.com%3E I guess my problem is also similar to the fact that my writes are well distributed and at a given time I could be writing to a lot of regions. Some of the regions receive very little data but since the flush algorithm choose at random what to flush when too many hlogs is hit, it will flush a region with less than 10mb of data causing too many small files. This in-turn causes compaction storms where even though major compactions is disabled, some of the minor get upgraded to major and that's when things start getting worse. My compaction queues are still the same and so I doubt I will be coming out of this storm without bumping up max hlogs for now. Reducing regions per server is one option but then I will be wasting my resources since the servers at current load are at 30% CPU and 25% RAM. Maybe I can bump up heap space and give more memory to the the memstore. Sorry, I am just thinking out loud. Thanks, Viral On Thu, Jun 27, 2013 at 2:40 PM, Jean-Daniel Cryans jdcry...@apache.orgwrote: No, all your data eventually makes it into the log, just potentially not as quickly :)
Re: 答复: flushing + compactions after config change
Hi Viral, the following are all needed for CMS: -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=0 -XX:+CMSClassUnloadingEnabled -XX:CMSMaxAbortablePrecleanTime=300 -XX:+CMSScavengeBeforeRemark and if your JDK version is greater than 1.6.23, then add : -XX:+UseCompressedOops -XX:SoftRefLRUPolicyMSPerMB=0 and you'd better add GC log -verbose:gc -XX:+PrintGCDateStamps -XX:+PrintGCDetails -Xloggc:$HBASE_LOG_DIR/gc.log if your JDK version is greater than 1.6.23, then add : -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=9 -XX:GCLogFileSize=20m Hope that helpful. On Fri, Jun 28, 2013 at 5:06 AM, Viral Bajaria viral.baja...@gmail.comwrote: Thanks Azuryy. Look forward to it. Does DEFERRED_LOG_FLUSH impact the number of WAL files that will be created ? Tried looking around but could not find the details. On Thu, Jun 27, 2013 at 7:53 AM, Azuryy Yu azury...@gmail.com wrote: your JVM options arenot enough. I will give you some detail when I go back office tomorrow. --Send from my Sony mobile.
