One issue you may run into is that 0.20 doesn't have https://issues.apache.org/jira/browse/HBASE-2066
a dev preview of 0.21 which does include that, and does improve performance should be available soon. On Wed, Jun 9, 2010 at 5:55 PM, Jinsong Hu <[email protected]> wrote: > Yes, I have done all the suggestion of the > http://wiki.apache.org/hadoop/PerformanceTuning. > > I just restarted the hbase cluster and recreated the table, the data > insertion looks fine for now and > I am getting about 1k record/second . I consider that to be reasonable > giving that my record is about > 10k bytes per record. but this is the beginning of the writing and I notice > that when the table is small, > the hbase works fine. when there are lots of records in the table already, > problem begin to happen. > I will report back and see how it goes after some more time. > > Jimmy. > > > -------------------------------------------------- > From: "Ryan Rawson" <[email protected]> > Sent: Wednesday, June 09, 2010 5:20 PM > To: <[email protected]> > Subject: Re: ideas to improve throughput of the base writting > >> I am not familiar with that exception, I have not seen of it before... >> perhaps someone else has? >> >> And my 200k rows/sec is over 19 machines. It is the average over many >> hours. My calculation of row size might not match how much data was >> flowing to disk, but I think it isn't too far off. >> >> Unfortunately comparing raw disk speed in a trivial benchmark (such as >> hdparm -t is) doesn't tell us how absolute speed of HBase must >> perform. This is because HBase does much more work than a raw disk >> write benchmark -- doing so to maintain structure and sorting. We can >> say that 'faster disks = faster HBase performance'. >> >> From the log lines you have pasted it sounds like the regionserver's >> flush ability is not keeping up with your rate of data input. How big >> are your records? What is your target input speed? Have you done >> anything on this page: >> http://wiki.apache.org/hadoop/PerformanceTuning >> >> >> >> On Wed, Jun 9, 2010 at 4:58 PM, Jinsong Hu <[email protected]> wrote: >>> >>> My hardware has 2 disks. I did a file copy on the machine and found that >>> I >>> can get 300 mbyte/second. >>> >>> At this time, I see my insertion is less than 1k/second. my row size is . >>> in >>> terms of disk writing. my record >>> insertion rate is far less than the hardware limit. my row size is about >>> 10K byte >>> >>> if in your i7-based server, you are doing 200k row/sec, each row is 200 >>> byte, then you are doing 40M byte/sec. >>> >>> in my case, if it behaves normally, I can get 100 row/sec * 10K byte =1M >>> /sec. >>> that is far from the disk speed. occasionally I can see 1k row/second. >>> which >>> is more reasonable in my case, >>> but I rarely get that performance. >>> >>> even worse, with the change done, now I have seem lots of compaction >>> failure: >>> >>> 2010-06-09 23:40:55,117 ERROR >>> org.apache.hadoop.hbase.regionserver.CompactSplitT >>> hread: Compaction failed for region Spam_MsgEventTable,2010-06-09 >>> 20:05:20\x0905 >>> 860d4bf1cb268ef69391cf97de9f64,1276121160527 >>> java.lang.RuntimeException: java.io.IOException: Could not find target >>> position >>> 65588 >>> at >>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileS >>> canner.java:61) >>> at >>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.j >>> ava:79) >>> at >>> org.apache.hadoop.hbase.regionserver.MinorCompactingStoreScanner.next >>> (MinorCompactingStoreScanner.java:96) >>> at >>> org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:920) >>> at >>> org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:764) >>> at >>> org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.ja >>> va:832) >>> at >>> org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.ja >>> va:785) >>> at >>> org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSp >>> litThread.java:93) >>> Caused by: java.io.IOException: Could not find target position 65588 >>> at >>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.fetchBlockAt(DFSClien >>> t.java:1556) >>> at >>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient >>> .java:1666) >>> at >>> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1 >>> 780) >>> at java.io.DataInputStream.read(DataInputStream.java:132) >>> at >>> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(Bou >>> ndedRangeFileInputStream.java:105) >>> at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100) >>> at >>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1 >>> 018) >>> at >>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:96 >>> 6) >>> at >>> org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java >>> :1159) >>> at >>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileS >>> canner.java:58) >>> ... 7 more >>> >>> I can't stop this unless I restarted the regionserver. After restart I >>> truncate the table, and when I list the table again in shell, >>> it appears 2 times. now I can't even disable the table and drop it. >>> >>> I will restart the whole hbase cluster and report the progress. >>> >>> Jimmy/ >>> >>> >>> >>> >>> >>> -------------------------------------------------- >>> From: "Ryan Rawson" <[email protected]> >>> Sent: Wednesday, June 09, 2010 4:16 PM >>> To: <[email protected]> >>> Subject: Re: ideas to improve throughput of the base writting >>> >>>> Hey, >>>> >>>> Sounds like you are hitting limits of your hardware... I dont think >>>> you mentioned the hardware spec you are running in this thread... >>>> >>>> What you are seeing is essentially the limits of HDFS's ability to >>>> take writes. The errors might be due to various HDFS setup problems >>>> (eg: xceiver count, file handle count, all outlined in various HBase >>>> "startup" docs)... But the overall performance might be limited by >>>> your hardware. >>>> >>>> For example, I use i7-based servers with 4 disks. This gives a >>>> reasonable IO bandwidth, and can cope with high rates of inserts (upto >>>> 100-200k rows/sec (each row is ~ 100-300 bytes). If you are running a >>>> 1 or 2 disk system it is possible you are hitting limits of what your >>>> hardware can do. >>>> >>>> Also note that the write-pipeline performance is ultimately defined in >>>> bytes/sec, not just 'rows/sec'... thus my rows were small, and if >>>> yours are big then you might be hitting a lower 'row/sec' limit even >>>> though the amount of bytes you are writing is higher than what i might >>>> have been doing. >>>> >>>> >>>> >>>> On Wed, Jun 9, 2010 at 3:59 PM, Jinsong Hu <[email protected]> >>>> wrote: >>>>> >>>>> I still get lots of repetition of >>>>> >>>>> 2010-06-09 22:54:38,428 WARN >>>>> org.apache.hadoop.hbase.regionserver.MemStoreFlushe >>>>> r: Region Spam_MsgEventTable,2010-06-09 >>>>> 20:05:20\x0905860d4bf1cb268ef69391cf97de >>>>> 9f64,1276121160527 has too many store files, putting it back at the end >>>>> of >>>>> the f >>>>> lush queue. >>>>> 2010-06-09 22:54:38,428 DEBUG >>>>> org.apache.hadoop.hbase.regionserver.CompactSplitT >>>>> hread: Compaction requested for region Spam_MsgEventTable,2010-06-09 >>>>> 20:05:20\x0 >>>>> 905860d4bf1cb268ef69391cf97de9f64,1276121160527/1537478401 because: >>>>> regionserver >>>>> /10.110.8.88:60020.cacheFlusher >>>>> >>>>> >>>>> I also saw lots of >>>>> >>>>> 2010-06-09 22:50:12,527 INFO >>>>> org.apache.hadoop.hbase.regionserver.HRegion: >>>>> Block >>>>> ing updates for 'IPC Server handler 1 on 60020' on region >>>>> Spam_MsgEventTable,201 >>>>> 0-06-09 20:05:20\x0905860d4bf1cb268ef69391cf97de9f64,1276121160527: >>>>> memstore >>>>> siz >>>>> e 512.0m is >= than blocking 512.0m size >>>>> 2010-06-09 22:50:12,598 INFO >>>>> org.apache.hadoop.hbase.regionserver.HRegion: >>>>> Block >>>>> ing updates for 'IPC Server handler 5 on 60020' on region >>>>> Spam_MsgEventTable,201 >>>>> 0-06-09 20:05:20\x0905860d4bf1cb268ef69391cf97de9f64,1276121160527: >>>>> memstore >>>>> siz >>>>> e 512.0m is >= than blocking 512.0m size >>>>> >>>>> even with the changed config. the regionserver has 4G ram. what else >>>>> can >>>>> be >>>>> wrong ? >>>>> >>>>> The insertion rate is still not good. >>>>> >>>>> Jimmy. >>>>> >>>>> >>>>> -------------------------------------------------- >>>>> From: "Jinsong Hu" <[email protected]> >>>>> Sent: Wednesday, June 09, 2010 1:59 PM >>>>> To: <[email protected]> >>>>> Subject: Re: ideas to improve throughput of the base writting >>>>> >>>>>> Thanks. I will make this change: >>>>>> >>>>>> <property> >>>>>> <name>hbase.hregion.memstore.block.multiplier</name> >>>>>> <value>8</value> >>>>>> </property> >>>>>> >>>>>> <property> >>>>>> <name>hbase.regionserver.msginterval</name> >>>>>> <value>10000</value> >>>>>> </property> >>>>>> >>>>>> <property> >>>>>> <name>hbase.hstore.compactionThreshold</name> >>>>>> <value>6</value> >>>>>> </property> >>>>>> >>>>>> >>>>>> <property> >>>>>> <name>hbase.hstore.blockingStoreFiles</name> >>>>>> <value>18</value> >>>>>> </property> >>>>>> >>>>>> >>>>>> and see how it goes. >>>>>> >>>>>> >>>>>> Jimmy. >>>>>> >>>>>> -------------------------------------------------- >>>>>> From: "Ryan Rawson" <[email protected]> >>>>>> Sent: Wednesday, June 09, 2010 1:49 PM >>>>>> To: <[email protected]> >>>>>> Subject: Re: ideas to improve throughput of the base writting >>>>>> >>>>>>> More background here... you are running into a situation where the >>>>>>> regionserver cannot flush fast enough and the size of the region's >>>>>>> memstore has climbed too high and thus you get that error message. >>>>>>> HBase attempts to protect itself by holding up clients (thus causing >>>>>>> the low performance you see). By expanding how big a memstore can >>>>>>> get >>>>>>> during times of stress you can improve performance, at the cost of >>>>>>> memory usage. That is what that setting is about. >>>>>>> >>>>>>> As for the 1.5 minute setting, that is the maximal amount of time a >>>>>>> handler thread will block for. You shouldn't need to tweak that >>>>>>> value, and reducing it could cause issues. >>>>>>> >>>>>>> Now, as for compacting, HBase will compact small files into larger >>>>>>> files, and on a massive upload you can expect to see this happen >>>>>>> constantly, thus tying up 1 cpu worth on your regionserver. You >>>>>>> could >>>>>>> potentially reduce that by increasing the value: >>>>>>> >>>>>>> <property> >>>>>>> <name>hbase.hstore.compactionThreshold</name> >>>>>>> <value>3</value> >>>>>>> >>>>>>> the value is interpreted as "if there are more than 3 files for a >>>>>>> region then run the compaction check". By raising this limit you can >>>>>>> accumulate more files before compacting them, thus reducing the >>>>>>> frequency of compactions but also potentially increasing the >>>>>>> performance of reads (more files to read = more seeks = slower). I'd >>>>>>> consider setting it to 5-7 or so in concert with setting >>>>>>> "hbase.hstore.blockingStoreFiles" to a value at least 2x that. >>>>>>> >>>>>>> All of these settings increase the amount of ram your regionserver >>>>>>> may >>>>>>> need, so you will want to ensure you have at least 4000m of ram set >>>>>>> in >>>>>>> hbase-env.sh. This is why they are set so conservatively in the >>>>>>> default shipping config. >>>>>>> >>>>>>> These are the 3 important settings that control how often compactions >>>>>>> occur and how RPC threads get blocked. Try tweaking all of them and >>>>>>> let me know if you are doing better. >>>>>>> >>>>>>> -ryan >>>>>>> >>>>>>> >>>>>>> On Wed, Jun 9, 2010 at 1:37 PM, Ryan Rawson <[email protected]> >>>>>>> wrote: >>>>>>>> >>>>>>>> you also want this config: >>>>>>>> >>>>>>>> <property> >>>>>>>> <name>hbase.hregion.memstore.block.multiplier</name> >>>>>>>> <value>8</value> >>>>>>>> </property> >>>>>>>> >>>>>>>> >>>>>>>> that should hopefully clear things up. >>>>>>>> >>>>>>>> -ryan >>>>>>>> >>>>>>>> On Wed, Jun 9, 2010 at 1:34 PM, Jinsong Hu <[email protected]> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> I checked the log, there are lots of >>>>>>>>> >>>>>>>>> e 128.1m is >= than blocking 128.0m size >>>>>>>>> 2010-06-09 17:26:36,736 INFO >>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegion: >>>>>>>>> Block >>>>>>>>> ing updates for 'IPC Server handler 8 on 60020' on region >>>>>>>>> Spam_MsgEventTable,201 >>>>>>>>> 0-06-09 05:25:32\x09c873847edf6e5390477494956ec04729,1276104002262: >>>>>>>>> memstore >>>>>>>>> siz >>>>>>>>> e 128.1m is >= than blocking 128.0m size >>>>>>>>> >>>>>>>>> then after that there are lots of >>>>>>>>> >>>>>>>>> 2010-06-09 17:26:36,800 DEBUG >>>>>>>>> org.apache.hadoop.hbase.regionserver.Store: >>>>>>>>> Added >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> hdfs://namenodes1.cloud.ppops.net:8020/hbase/Spam_MsgEventTable/376337880/messag >>>>>>>>> e_compound_terms/7606939244559826252, entries=30869, >>>>>>>>> sequenceid=8350447892, >>>>>>>>> mems >>>>>>>>> ize=7.2m, filesize=3.4m to Spam_MsgEventTable,2010-06-09 >>>>>>>>> 05:25:32\x09c873847edf6 >>>>>>>>> >>>>>>>>> >>>>>>>>> then lots of >>>>>>>>> >>>>>>>>> 2010-06-09 17:26:39,005 INFO >>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegion: >>>>>>>>> Unblo >>>>>>>>> cking updates for region Spam_MsgEventTable,2010-06-09 >>>>>>>>> 05:25:32\x09c873847edf6e5 >>>>>>>>> 390477494956ec04729,1276104002262 'IPC Server handler 8 on 60020' >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> This cycle happens again and again in the log. What can I do in >>>>>>>>> this >>>>>>>>> case >>>>>>>>> to speed up writing ? >>>>>>>>> right now the writing speed is really slow. close to 4 rows/second >>>>>>>>> for >>>>>>>>> a >>>>>>>>> regionserver. >>>>>>>>> >>>>>>>>> I checked the code and try to find out why there are so many store >>>>>>>>> files, >>>>>>>>> and I noticed each second >>>>>>>>> the regionserver reports to master, it calls the memstore flush and >>>>>>>>> write a >>>>>>>>> store file. >>>>>>>>> >>>>>>>>> the parameter hbase.regionserver.msginterval default value is 1 >>>>>>>>> second. >>>>>>>>> I am >>>>>>>>> thinking to change to 10 second. >>>>>>>>> can that help ? I am also thinking to change >>>>>>>>> hbase.hstore.blockingStoreFiles >>>>>>>>> to 1000. I noticed that there is a parameter >>>>>>>>> hbase.hstore.blockingWaitTime with default value of 1.5 minutes. as >>>>>>>>> long as >>>>>>>>> the 1.5 minutes is reached, >>>>>>>>> the compaction is executed. I am fine with running compaction every >>>>>>>>> 1.5 >>>>>>>>> minutes, but running compaction every second >>>>>>>>> and causing CPU consistently higher than 100% is not wanted. >>>>>>>>> >>>>>>>>> Any suggestion what kind of parameters to change to improve my >>>>>>>>> writing >>>>>>>>> speed >>>>>>>>> ? >>>>>>>>> >>>>>>>>> Jimmy >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -------------------------------------------------- >>>>>>>>> From: "Ryan Rawson" <[email protected]> >>>>>>>>> Sent: Wednesday, June 09, 2010 1:01 PM >>>>>>>>> To: <[email protected]> >>>>>>>>> Subject: Re: ideas to improve throughput of the base writting >>>>>>>>> >>>>>>>>>> The log will say something like "blocking updates to..." when you >>>>>>>>>> hit >>>>>>>>>> a limit. That log you indicate is just the regionserver >>>>>>>>>> attempting >>>>>>>>>> to >>>>>>>>>> compact a region, but shouldn't prevent updates. >>>>>>>>>> >>>>>>>>>> what else does your logfile say? Search for the string (case >>>>>>>>>> insensitive) "blocking updates"... >>>>>>>>>> >>>>>>>>>> -ryan >>>>>>>>>> >>>>>>>>>> On Wed, Jun 9, 2010 at 11:52 AM, Jinsong Hu >>>>>>>>>> <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> I made this change >>>>>>>>>>> <property> >>>>>>>>>>> <name>hbase.hstore.blockingStoreFiles</name> >>>>>>>>>>> <value>15</value> >>>>>>>>>>> </property> >>>>>>>>>>> >>>>>>>>>>> the system is still slow. >>>>>>>>>>> >>>>>>>>>>> Here is the most recent value for the region : >>>>>>>>>>> stores=21, storefiles=186, storefileSizeMB=9681, >>>>>>>>>>> memstoreSizeMB=128, >>>>>>>>>>> storefileIndexSizeMB=12 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> And the same log still happens: >>>>>>>>>>> >>>>>>>>>>> 2010-06-09 18:36:40,577 WARN org.apache.h >>>>>>>>>>> adoop.hbase.regionserver.MemStoreFlusher: Region >>>>>>>>>>> SOME_ABCEventTable,2010-06-09 0 >>>>>>>>>>> 9:56:56\x093dc01b4d2c4872963717d80d8b5c74b1,1276107447570 has too >>>>>>>>>>> many >>>>>>>>>>> store >>>>>>>>>>> fil >>>>>>>>>>> es, putting it back at the end of the flush queue. >>>>>>>>>>> >>>>>>>>>>> One idea that I have now is to further increase the >>>>>>>>>>> hbase.hstore.blockingStoreFiles to a very high >>>>>>>>>>> Number, such as 1000. What is the negative impact of this change >>>>>>>>>>> ? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Jimmy >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -------------------------------------------------- >>>>>>>>>>> From: "Ryan Rawson" <[email protected]> >>>>>>>>>>> Sent: Monday, June 07, 2010 3:58 PM >>>>>>>>>>> To: <[email protected]> >>>>>>>>>>> Subject: Re: ideas to improve throughput of the base writting >>>>>>>>>>> >>>>>>>>>>>> Try setting this config value: >>>>>>>>>>>> >>>>>>>>>>>> <property> >>>>>>>>>>>> <name>hbase.hstore.blockingStoreFiles</name> >>>>>>>>>>>> <value>15</value> >>>>>>>>>>>> </property> >>>>>>>>>>>> >>>>>>>>>>>> and see if that helps. >>>>>>>>>>>> >>>>>>>>>>>> The thing about the 1 compact thread is the scarce resources >>>>>>>>>>>> being >>>>>>>>>>>> preserved in this case is cluster IO. People have had issues >>>>>>>>>>>> with >>>>>>>>>>>> compaction IO being too heavy. >>>>>>>>>>>> >>>>>>>>>>>> in your case, this setting can let the regionserver build up >>>>>>>>>>>> more >>>>>>>>>>>> store files without pausing your import. >>>>>>>>>>>> >>>>>>>>>>>> -ryan >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Jun 7, 2010 at 3:52 PM, Jinsong Hu >>>>>>>>>>>> <[email protected]> >>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Hi, There: >>>>>>>>>>>>> While saving lots of data to on hbase, I noticed that the >>>>>>>>>>>>> regionserver >>>>>>>>>>>>> CPU >>>>>>>>>>>>> went to more than 100%. examination shows that the hbase >>>>>>>>>>>>> CompactSplit >>>>>>>>>>>>> is >>>>>>>>>>>>> spending full time working on compacting/splitting hbase store >>>>>>>>>>>>> files. >>>>>>>>>>>>> The >>>>>>>>>>>>> machine I have is an 8 core machine. because there is only one >>>>>>>>>>>>> comact/split >>>>>>>>>>>>> thread in hbase, only one core is fully used. >>>>>>>>>>>>> I continue to submit map/reduce job to insert records to >>>>>>>>>>>>> hbase. >>>>>>>>>>>>> most >>>>>>>>>>>>> of >>>>>>>>>>>>> the time, the job runs very fast, around 1-5 minutes. But >>>>>>>>>>>>> occasionally, >>>>>>>>>>>>> it >>>>>>>>>>>>> can take 2 hours. That is very bad to me. I highly suspect that >>>>>>>>>>>>> the >>>>>>>>>>>>> occasional slow insertion is related to the >>>>>>>>>>>>> insufficient speed compactsplit thread. >>>>>>>>>>>>> I am thinking that I should parallize the compactsplit thread, >>>>>>>>>>>>> the >>>>>>>>>>>>> code >>>>>>>>>>>>> has >>>>>>>>>>>>> this : the for loop "for (Store store: stores.values()) " can >>>>>>>>>>>>> be >>>>>>>>>>>>> parallized via java 5's threadpool , thus multiple cores are >>>>>>>>>>>>> used >>>>>>>>>>>>> instead >>>>>>>>>>>>> only one core is used. I wonder if this will help to increase >>>>>>>>>>>>> the >>>>>>>>>>>>> throughput. >>>>>>>>>>>>> >>>>>>>>>>>>> Somebody mentioned that I can increase the regionsize to that >>>>>>>>>>>>> I >>>>>>>>>>>>> don't >>>>>>>>>>>>> do >>>>>>>>>>>>> so >>>>>>>>>>>>> many compaction. Under heavy writing situation. >>>>>>>>>>>>> does anybody have experience showing it helps ? >>>>>>>>>>>>> >>>>>>>>>>>>> Jimmy. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> byte [] compactStores(final boolean majorCompaction) >>>>>>>>>>>>> >>>>>>>>>>>>> throws IOException { >>>>>>>>>>>>> >>>>>>>>>>>>> if (this.closing.get() || this.closed.get()) { >>>>>>>>>>>>> >>>>>>>>>>>>> LOG.debug("Skipping compaction on " + this + " because >>>>>>>>>>>>> closing/closed"); >>>>>>>>>>>>> >>>>>>>>>>>>> return null; >>>>>>>>>>>>> >>>>>>>>>>>>> } >>>>>>>>>>>>> >>>>>>>>>>>>> splitsAndClosesLock.readLock().lock(); >>>>>>>>>>>>> >>>>>>>>>>>>> try { >>>>>>>>>>>>> >>>>>>>>>>>>> byte [] splitRow = null; >>>>>>>>>>>>> >>>>>>>>>>>>> if (this.closed.get()) { >>>>>>>>>>>>> >>>>>>>>>>>>> return splitRow; >>>>>>>>>>>>> >>>>>>>>>>>>> } >>>>>>>>>>>>> >>>>>>>>>>>>> try { >>>>>>>>>>>>> >> >
