Re: HBase Random Read latency 100ms
Hi All, I just ran only 8 parallel clients, With SCR Enabled : Average Latency is 80 ms, IO Wait % is around 8 With SCR Disabled: Average Latency is 40 ms, IO Wait % is around 2 I always thought SCR enabled, allows a client co-located with the DataNode to read HDFS file blocks directly. This gives a performance boost to distributed clients that are aware of locality. Is my understanding wrong OR it doesn't apply to my scenario? Meanwhile I will try setting the parameter suggested by Lars and post you the results. Thanks, Ramu On Wed, Oct 9, 2013 at 2:29 PM, lars hofhansl la...@apache.org wrote: Good call. Could try to enable hbase.regionserver.checksum.verify, which will cause HBase to do its own checksums rather than relying on HDFS (and which saves 1 IO per block get). I do think you can expect the index blocks to be cached at all times. -- Lars From: Vladimir Rodionov vrodio...@carrieriq.com To: user@hbase.apache.org user@hbase.apache.org Sent: Tuesday, October 8, 2013 8:44 PM Subject: RE: HBase Random Read latency 100ms Upd. Each HBase Get = 2 HDFS read IO (index block + data block)= 4 File IO (data + .crc) in a worst case. I think if Bloom Filter is enabled than it is going to be 6 File IO in a worst case (large data set), therefore you will have not 5 IO requests in queue but up to 20-30 IO requests in a queue This definitely explains 100ms avg latency. Best regards, Vladimir Rodionov Principal Platform Engineer Carrier IQ, www.carrieriq.com e-mail: vrodio...@carrieriq.com From: Vladimir Rodionov Sent: Tuesday, October 08, 2013 7:24 PM To: user@hbase.apache.org Subject: RE: HBase Random Read latency 100ms Ramu, You have 8 server boxes and 10 client. You have 40 requests in parallel - 5 per RS/DN? You have 5 requests on random reads in a IO queue of your single RAID1. With avg read latency of 10 ms, 5 requests in queue will give us 30ms. Add some overhead of HDFS + HBase and you will probably have your issue explained ? Your bottleneck is your disk system, I think. When you serve most of requests from disks as in your large data set scenario, make sure you have adequate disk sub-system and that it is configured properly. Block Cache and OS page can not help you in this case as working data set is larger than both caches. Good performance numbers in small data set scenario are explained by the fact that data fits into OS page cache and Block Cache - you do not read data from disk even if you disable block cache. Best regards, Vladimir Rodionov Principal Platform Engineer Carrier IQ, www.carrieriq.com e-mail: vrodio...@carrieriq.com From: Ramu M S [ramu.ma...@gmail.com] Sent: Tuesday, October 08, 2013 6:00 PM To: user@hbase.apache.org Subject: Re: HBase Random Read latency 100ms Hi All, After few suggestions from the mails earlier I changed the following, 1. Heap Size to 16 GB 2. Block Size to 16KB 3. HFile size to 8 GB (Table now has 256 regions, 32 per server) 4. Data Locality Index is 100 in all RS I have clients running in 10 machines, each with 4 threads. So total 40. This is same in all tests. Result: 1. Average latency is still 100ms. 2. Heap occupancy is around 2-2.5 GB in all RS Few more tests carried out yesterday, TEST 1: Small data set (100 Million records, each with 724 bytes). === Configurations: 1. Heap Size to 1 GB 2. Block Size to 16KB 3. HFile size to 1 GB (Table now has 128 regions, 16 per server) 4. Data Locality Index is 100 in all RS I disabled Block Cache on the table, to make sure I read everything from disk, most of the time. Result: 1. Average Latency is 8ms and throughput went up to 6K/Sec per RS. 2. With Block Cache enabled again, I got average latency around 2ms and throughput of 10K/Sec per RS. Heap occupancy around 650 MB 3. Increased the Heap to 16GB, with Block Cache still enabled, I got average latency around 1 ms and throughput 20K/Sec per RS Heap Occupancy around 2-2.5 GB in all RS TEST 2: Large Data set (1.8 Billion records, each with 724 bytes) == Configurations: 1. Heap Size to 1 GB 2. Block Size to 16KB 3. HFile size to 1 GB (Table now has 2048 regions, 256 per server) 4. Data Locality Index is 100 in all RS Result: 1. Average Latency is 500ms to start with and gradually decreases, but even after around 100 Million reads it is still 100 ms 2. Block Cache = TRUE/FALSE does not make any difference here. Even Heap Size (1GB / 16GB) does not make any difference. 3. Heap occupancy is around 2-2.5 GB under 16GB Heap and around 650 MB under 1GB Heap. GC Time in all of the scenarios is around 2ms/Second, as shown in the Cloudera Manager. Reading most of the items from Disk in
Re: HBase Random Read latency 100ms
Hi All, Sorry. There was some mistake in the tests (Clients were not reduced, forgot to change the parameter before running tests). With 8 Clients and, SCR Enabled : Average Latency is 25 ms, IO Wait % is around 8 SCR Disabled: Average Latency is 10 ms, IO Wait % is around 2 Still, SCR disabled gives better results, which confuse me. Can anyone clarify? Also, I tried setting the parameter (hbase.regionserver.checksum.verify as true) Lars suggested with SCR disabled. Average Latency is around 9.8 ms, a fraction lesser. Thanks Ramu On Wed, Oct 9, 2013 at 3:32 PM, Ramu M S ramu.ma...@gmail.com wrote: Hi All, I just ran only 8 parallel clients, With SCR Enabled : Average Latency is 80 ms, IO Wait % is around 8 With SCR Disabled: Average Latency is 40 ms, IO Wait % is around 2 I always thought SCR enabled, allows a client co-located with the DataNode to read HDFS file blocks directly. This gives a performance boost to distributed clients that are aware of locality. Is my understanding wrong OR it doesn't apply to my scenario? Meanwhile I will try setting the parameter suggested by Lars and post you the results. Thanks, Ramu On Wed, Oct 9, 2013 at 2:29 PM, lars hofhansl la...@apache.org wrote: Good call. Could try to enable hbase.regionserver.checksum.verify, which will cause HBase to do its own checksums rather than relying on HDFS (and which saves 1 IO per block get). I do think you can expect the index blocks to be cached at all times. -- Lars From: Vladimir Rodionov vrodio...@carrieriq.com To: user@hbase.apache.org user@hbase.apache.org Sent: Tuesday, October 8, 2013 8:44 PM Subject: RE: HBase Random Read latency 100ms Upd. Each HBase Get = 2 HDFS read IO (index block + data block)= 4 File IO (data + .crc) in a worst case. I think if Bloom Filter is enabled than it is going to be 6 File IO in a worst case (large data set), therefore you will have not 5 IO requests in queue but up to 20-30 IO requests in a queue This definitely explains 100ms avg latency. Best regards, Vladimir Rodionov Principal Platform Engineer Carrier IQ, www.carrieriq.com e-mail: vrodio...@carrieriq.com From: Vladimir Rodionov Sent: Tuesday, October 08, 2013 7:24 PM To: user@hbase.apache.org Subject: RE: HBase Random Read latency 100ms Ramu, You have 8 server boxes and 10 client. You have 40 requests in parallel - 5 per RS/DN? You have 5 requests on random reads in a IO queue of your single RAID1. With avg read latency of 10 ms, 5 requests in queue will give us 30ms. Add some overhead of HDFS + HBase and you will probably have your issue explained ? Your bottleneck is your disk system, I think. When you serve most of requests from disks as in your large data set scenario, make sure you have adequate disk sub-system and that it is configured properly. Block Cache and OS page can not help you in this case as working data set is larger than both caches. Good performance numbers in small data set scenario are explained by the fact that data fits into OS page cache and Block Cache - you do not read data from disk even if you disable block cache. Best regards, Vladimir Rodionov Principal Platform Engineer Carrier IQ, www.carrieriq.com e-mail: vrodio...@carrieriq.com From: Ramu M S [ramu.ma...@gmail.com] Sent: Tuesday, October 08, 2013 6:00 PM To: user@hbase.apache.org Subject: Re: HBase Random Read latency 100ms Hi All, After few suggestions from the mails earlier I changed the following, 1. Heap Size to 16 GB 2. Block Size to 16KB 3. HFile size to 8 GB (Table now has 256 regions, 32 per server) 4. Data Locality Index is 100 in all RS I have clients running in 10 machines, each with 4 threads. So total 40. This is same in all tests. Result: 1. Average latency is still 100ms. 2. Heap occupancy is around 2-2.5 GB in all RS Few more tests carried out yesterday, TEST 1: Small data set (100 Million records, each with 724 bytes). === Configurations: 1. Heap Size to 1 GB 2. Block Size to 16KB 3. HFile size to 1 GB (Table now has 128 regions, 16 per server) 4. Data Locality Index is 100 in all RS I disabled Block Cache on the table, to make sure I read everything from disk, most of the time. Result: 1. Average Latency is 8ms and throughput went up to 6K/Sec per RS. 2. With Block Cache enabled again, I got average latency around 2ms and throughput of 10K/Sec per RS. Heap occupancy around 650 MB 3. Increased the Heap to 16GB, with Block Cache still enabled, I got average latency around 1 ms and throughput 20K/Sec per RS Heap Occupancy around 2-2.5 GB in all RS TEST 2: Large Data set (1.8 Billion records, each with 724 bytes) == Configurations: 1. Heap Size to 1
Re: exportSnapshot MISSING_FILES
Would this question be better suited for the developer mailing list? Adrian On Tue, Oct 8, 2013 at 7:01 PM, Adrian Sandulescu sandulescu.adr...@gmail.com wrote: Also, here are the files in S3: $ hadoop fs -ls s3n://AKIAIWNEBAESDM4DKBGA:UEFai8K1IBrjOKlXB2hbIbTFDJ8apubuA01LCc02@hbase-export /hbase/.archive/campaign_digital_ad_time_dev/d9e6cff519bdd232d7a7f8eb676d92ee/v/6912d21c32754e7cb34ac85952a67ce1 Found 1 items -rwxrwxrwx 1 741047906 2013-10-08 13:45 s3n://AKIAIWNEBAESDM4DKBGA:UEFai8K1IBrjOKlXB2hbIbTFDJ8apubuA01LCc02@hbase-export /hbase/.archive/campaign_digital_ad_time_dev/d9e6cff519bdd232d7a7f8eb676d92ee/v/6912d21c32754e7cb34ac85952a67ce1 $ hadoop fs -ls s3://AKIAIWNEBAESDM4DKBGA:UEFai8K1IBrjOKlXB2hbIbTFDJ8apubuA01LCc02@hbase-export /hbase/.archive/campaign_digital_ad_time_dev/d9e6cff519bdd232d7a7f8eb676d92ee/v/6912d21c32754e7cb34ac85952a67ce1 Found 1 items -rwxrwxrwx 1 741047906 1970-01-01 00:00 s3://AKIAIWNEBAESDM4DKBGA:UEFai8K1IBrjOKlXB2hbIbTFDJ8apubuA01LCc02@hbase-export /hbase/.archive/campaign_digital_ad_time_dev/d9e6cff519bdd232d7a7f8eb676d92ee/v/6912d21c32754e7cb34ac85952a67ce1 Thank you, Adrian On Tue, Oct 8, 2013 at 6:44 PM, Adrian Sandulescu sandulescu.adr...@gmail.com wrote: Yes, I was just digging. From a successful s3n:// import 2013-10-08 14:57:04,816 INFO org.apache.hadoop.hbase.snapshot.ExportSnapshot: copy file input=v/campaign_digital_ad_time_dev=d9e6cff519bdd232d7a7f8eb676d92ee-6912d21c32754e7cb34ac85952a67ce1 output=hdfs://mycluster:8020/hbase/.archive/campaign_digital_ad_time_dev/d9e6cff519bdd232d7a7f8eb676d92ee/v/6912d21c32754e7cb34ac85952a67ce1 2013-10-08 14:57:04,965 INFO org.apache.hadoop.fs.s3native.NativeS3FileSystem: Opening 's3n://AKIAIWNEBAESDM4DKBGA:UEFai8K1IBrjOKlXB2hbIbTFDJ8apubuA01LCc02@hbase-export/hbase/.archive/campaign_digital_ad_time_dev/d9e6cff519bdd232d7a7f8eb676d92ee/v/6912d21c32754e7cb34ac85952a67ce1' for reading 2013-10-08 14:57:05,039 INFO org.apache.hadoop.fs.s3native.NativeS3FileSystem: Opening key 'hbase/.archive/campaign_digital_ad_time_dev/d9e6cff519bdd232d7a7f8eb676d92ee/v/6912d21c32754e7cb34ac85952a67ce1' for reading at position '0' 2013-10-08 14:57:05,299 INFO org.apache.hadoop.hbase.snapshot.ExportSnapshot: Skip copy v/campaign_digital_ad_time_dev=d9e6cff519bdd232d7a7f8eb676d92ee-6912d21c32754e7cb34ac85952a67ce1 to hdfs://mycluster:8020/hbase/.archive/campaign_digital_ad_time_dev/d9e6cff519bdd232d7a7f8eb676d92ee/v/6912d21c32754e7cb34ac85952a67ce1, same file. 2013-10-08 14:57:05,300 INFO org.apache.hadoop.hbase.snapshot.ExportSnapshot: copy completed for input=v/campaign_digital_ad_time_dev=d9e6cff519bdd232d7a7f8eb676d92ee-6912d21c32754e7cb34ac85952a67ce1 output=hdfs://mycluster:8020/hbase/.archive/campaign_digital_ad_time_dev/d9e6cff519bdd232d7a7f8eb676d92ee/v/6912d21c32754e7cb34ac85952a67ce1 From a failed s3:// import 2013-10-08 15:27:21,810 INFO org.apache.hadoop.hbase.snapshot.ExportSnapshot: copy file input=v/campaign_digital_ad_time_dev=d9e6cff519bdd232d7a7f8eb676d92ee-6912d21c32754e7cb34ac85952a67ce1 output=hdfs://mycluster:8020/hbase/.archive/campaign_digital_ad_time_dev/d9e6cff519bdd232d7a7f8eb676d92ee/v/6912d21c32754e7cb34ac85952a67ce1 2013-10-08 15:27:21,834 ERROR org.apache.hadoop.hbase.snapshot.ExportSnapshot: Unable to open source file=v/campaign_digital_ad_time_dev=d9e6cff519bdd232d7a7f8eb676d92ee-6912d21c32754e7cb34ac85952a67ce1 java.io.IOException: No such file. at org.apache.hadoop.fs.s3.S3FileSystem.checkFile(S3FileSystem.java:181) at org.apache.hadoop.fs.s3.S3FileSystem.open(S3FileSystem.java:246) at org.apache.hadoop.hbase.io.FileLink$FileLinkInputStream.tryOpen(FileLink.java:289) at org.apache.hadoop.hbase.io.FileLink$FileLinkInputStream.init(FileLink.java:120) at org.apache.hadoop.hbase.io.FileLink$FileLinkInputStream.init(FileLink.java:111) at org.apache.hadoop.hbase.io.FileLink.open(FileLink.java:390) at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.openSourceFile(ExportSnapshot.java:302) at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.copyFile(ExportSnapshot.java:175) at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.map(ExportSnapshot.java:146) at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.map(ExportSnapshot.java:95) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330) at org.apache.hadoop.mapred.Child$4.run(Child.java:268) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) at org.apache.hadoop.mapred.Child.main(Child.java:262) Thank you, Adrian On Tue,
Re: Spatial data posting in HBase
Actually just to follow up.. Geohash kinda has some issues at the global scale. Edge conditions on quadrant boundaries. You're better off tiling the map, and then within each tile you can filter the list. I think you'd also want to be a bit more OO than what's in the HBase book. You will want to create a spatial point object that has x,y and z coordinates instead of storing the x, y, and z coordinates as separate columns. Depending on the size of the tile the number of coordinates that you have to filter can become sparse, such that its actually faster to just have a single thread run through it. Of course YMMV, however from experience... this works the best. HTH -Mike On Oct 7, 2013, at 7:20 PM, Michael Segel michael_se...@hotmail.com wrote: I don't think you really want to use geo hashes. Check out InfoQ and Search on Boris Lublinsky. On Oct 4, 2013, at 3:01 AM, Adrien Mogenet adrien.moge...@gmail.com wrote: If you mean insert and query spatial data, look at algorithms that are distributed databases compliant : geohash, z-index, voronoi diagram... Well, that makes me want to write a blog article about these topics :) On Tue, Sep 24, 2013 at 3:43 PM, Ted Yu yuzhih...@gmail.com wrote: There're plenty of examples in unit tests. e.g. : Put put = new Put(Bytes.toBytes(row + String.format(%1$04d, i))); put.add(family, null, value); table.put(put); value can be obtained through Bytes.toBytes(). table is an HTable. Cheers On Tue, Sep 24, 2013 at 4:15 AM, cto ankur...@tcs.com wrote: Hi , I am very new in HBase. Could you please let me know , how to insert spatial data (Latitude / Longitude) in HBase using Java . -- View this message in context: http://apache-hbase.679495.n3.nabble.com/Spatial-data-posting-in-HBase-tp4051123.html Sent from the HBase User mailing list archive at Nabble.com. -- Adrien Mogenet http://www.borntosegfault.com The opinions expressed here are mine, while they may reflect a cognitive thought, that is purely accidental. Use at your own risk. Michael Segel michael_segel (AT) hotmail.com The opinions expressed here are mine, while they may reflect a cognitive thought, that is purely accidental. Use at your own risk. Michael Segel michael_segel (AT) hotmail.com
Re: Spatial data posting in HBase
Consider using Solr, which provides a lot of geospatial search support. Otis Solr ElasticSearch Support http://sematext.com/ On Sep 24, 2013 8:29 AM, cto ankur...@tcs.com wrote: Hi , I am very new in HBase. Could you please let me know , how to insert spatial data (Latitude / Longitude) in HBase using Java . -- View this message in context: http://apache-hbase.679495.n3.nabble.com/Spatial-data-posting-in-HBase-tp4051123.html Sent from the HBase User mailing list archive at Nabble.com.
Re: Region is not online: -ROOT-,,0
Can you tell us what HBase and Hadoop versions you're using ? Thanks On Oct 9, 2013, at 5:32 AM, Venkata siva kamesh Bhallamudi kam.iit...@gmail.com wrote: Hi All, I am getting following exception when I am trying to run *hbck *command Exception in thread main org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3166) at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1986) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:384) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426) Please help me understanding this exception and how to fix this. Thanks Regards Kamesh.
Region is not online: -ROOT-,,0
Hi All, I am getting following exception when I am trying to run *hbck *command Exception in thread main org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3166) at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1986) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:384) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426) Please help me understanding this exception and how to fix this. Thanks Regards Kamesh.
Re: Spatial data posting in HBase
And Solr has what to do with storing data in HBase? I guess its true… if all you have is a hammer… The point I was raising was that geohash isn't the most efficient way to go when you look at the problem at a global level… On Oct 9, 2013, at 9:34 AM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Consider using Solr, which provides a lot of geospatial search support. Otis Solr ElasticSearch Support http://sematext.com/ On Sep 24, 2013 8:29 AM, cto ankur...@tcs.com wrote: Hi , I am very new in HBase. Could you please let me know , how to insert spatial data (Latitude / Longitude) in HBase using Java . -- View this message in context: http://apache-hbase.679495.n3.nabble.com/Spatial-data-posting-in-HBase-tp4051123.html Sent from the HBase User mailing list archive at Nabble.com. smime.p7s Description: S/MIME cryptographic signature
Re: Region is not online: -ROOT-,,0
Hi Kamesh The problem cannot be specifically isolated. But i suggest you can try restarting your master. But the exception says this In HBase there are catalog tables called ROOT and META(in 0.94.x) and only META in 0.96+. Basically META has information on all user regions and ROOT is a table that stores information on where the META table is. For the cluster to function these two tables should be online. If not the cluster would be unusable. Regards Ram On Wed, Oct 9, 2013 at 8:14 PM, Ted Yu yuzhih...@gmail.com wrote: Can you tell us what HBase and Hadoop versions you're using ? Thanks On Oct 9, 2013, at 5:32 AM, Venkata siva kamesh Bhallamudi kam.iit...@gmail.com wrote: Hi All, I am getting following exception when I am trying to run *hbck *command Exception in thread main org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3166) at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1986) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:384) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426) Please help me understanding this exception and how to fix this. Thanks Regards Kamesh.
RE: HBase Random Read latency 100ms
I can't say for SCR. There is a possibility that the feature is broken, of course. But the fact that hbase.regionserver.checksum.verify does not affect performance means that OS caches effectively HDFS checksum files. Best regards, Vladimir Rodionov Principal Platform Engineer Carrier IQ, www.carrieriq.com e-mail: vrodio...@carrieriq.com From: Ramu M S [ramu.ma...@gmail.com] Sent: Wednesday, October 09, 2013 12:11 AM To: user@hbase.apache.org; lars hofhansl Subject: Re: HBase Random Read latency 100ms Hi All, Sorry. There was some mistake in the tests (Clients were not reduced, forgot to change the parameter before running tests). With 8 Clients and, SCR Enabled : Average Latency is 25 ms, IO Wait % is around 8 SCR Disabled: Average Latency is 10 ms, IO Wait % is around 2 Still, SCR disabled gives better results, which confuse me. Can anyone clarify? Also, I tried setting the parameter (hbase.regionserver.checksum.verify as true) Lars suggested with SCR disabled. Average Latency is around 9.8 ms, a fraction lesser. Thanks Ramu On Wed, Oct 9, 2013 at 3:32 PM, Ramu M S ramu.ma...@gmail.com wrote: Hi All, I just ran only 8 parallel clients, With SCR Enabled : Average Latency is 80 ms, IO Wait % is around 8 With SCR Disabled: Average Latency is 40 ms, IO Wait % is around 2 I always thought SCR enabled, allows a client co-located with the DataNode to read HDFS file blocks directly. This gives a performance boost to distributed clients that are aware of locality. Is my understanding wrong OR it doesn't apply to my scenario? Meanwhile I will try setting the parameter suggested by Lars and post you the results. Thanks, Ramu On Wed, Oct 9, 2013 at 2:29 PM, lars hofhansl la...@apache.org wrote: Good call. Could try to enable hbase.regionserver.checksum.verify, which will cause HBase to do its own checksums rather than relying on HDFS (and which saves 1 IO per block get). I do think you can expect the index blocks to be cached at all times. -- Lars From: Vladimir Rodionov vrodio...@carrieriq.com To: user@hbase.apache.org user@hbase.apache.org Sent: Tuesday, October 8, 2013 8:44 PM Subject: RE: HBase Random Read latency 100ms Upd. Each HBase Get = 2 HDFS read IO (index block + data block)= 4 File IO (data + .crc) in a worst case. I think if Bloom Filter is enabled than it is going to be 6 File IO in a worst case (large data set), therefore you will have not 5 IO requests in queue but up to 20-30 IO requests in a queue This definitely explains 100ms avg latency. Best regards, Vladimir Rodionov Principal Platform Engineer Carrier IQ, www.carrieriq.com e-mail: vrodio...@carrieriq.com From: Vladimir Rodionov Sent: Tuesday, October 08, 2013 7:24 PM To: user@hbase.apache.org Subject: RE: HBase Random Read latency 100ms Ramu, You have 8 server boxes and 10 client. You have 40 requests in parallel - 5 per RS/DN? You have 5 requests on random reads in a IO queue of your single RAID1. With avg read latency of 10 ms, 5 requests in queue will give us 30ms. Add some overhead of HDFS + HBase and you will probably have your issue explained ? Your bottleneck is your disk system, I think. When you serve most of requests from disks as in your large data set scenario, make sure you have adequate disk sub-system and that it is configured properly. Block Cache and OS page can not help you in this case as working data set is larger than both caches. Good performance numbers in small data set scenario are explained by the fact that data fits into OS page cache and Block Cache - you do not read data from disk even if you disable block cache. Best regards, Vladimir Rodionov Principal Platform Engineer Carrier IQ, www.carrieriq.com e-mail: vrodio...@carrieriq.com From: Ramu M S [ramu.ma...@gmail.com] Sent: Tuesday, October 08, 2013 6:00 PM To: user@hbase.apache.org Subject: Re: HBase Random Read latency 100ms Hi All, After few suggestions from the mails earlier I changed the following, 1. Heap Size to 16 GB 2. Block Size to 16KB 3. HFile size to 8 GB (Table now has 256 regions, 32 per server) 4. Data Locality Index is 100 in all RS I have clients running in 10 machines, each with 4 threads. So total 40. This is same in all tests. Result: 1. Average latency is still 100ms. 2. Heap occupancy is around 2-2.5 GB in all RS Few more tests carried out yesterday, TEST 1: Small data set (100 Million records, each with 724 bytes). === Configurations: 1. Heap Size to 1 GB 2. Block Size to 16KB 3. HFile size to 1 GB (Table now has 128 regions, 16 per server) 4. Data Locality Index is 100 in all RS I disabled Block Cache on the table, to make sure I read everything from disk, most of the time.
Re: HBase Random Read latency 100ms
On Wed, Oct 9, 2013 at 10:59 AM, Vladimir Rodionov vrodio...@carrieriq.comwrote: I can't say for SCR. There is a possibility that the feature is broken, of course. But the fact that hbase.regionserver.checksum.verify does not affect performance means that OS caches effectively HDFS checksum files. See OS cache + SCR VS HBase CRC over OS cache+SCR in this document I shared some time ago: https://docs.google.com/spreadsheet/pub?key=0Ao87IrzZJSaydENaem5USWg4TlRKcHl0dEtTS2NBOUEoutput=html In an all-in-memory test it shows a pretty big difference. J-D Best regards, Vladimir Rodionov Principal Platform Engineer Carrier IQ, www.carrieriq.com e-mail: vrodio...@carrieriq.com From: Ramu M S [ramu.ma...@gmail.com] Sent: Wednesday, October 09, 2013 12:11 AM To: user@hbase.apache.org; lars hofhansl Subject: Re: HBase Random Read latency 100ms Hi All, Sorry. There was some mistake in the tests (Clients were not reduced, forgot to change the parameter before running tests). With 8 Clients and, SCR Enabled : Average Latency is 25 ms, IO Wait % is around 8 SCR Disabled: Average Latency is 10 ms, IO Wait % is around 2 Still, SCR disabled gives better results, which confuse me. Can anyone clarify? Also, I tried setting the parameter (hbase.regionserver.checksum.verify as true) Lars suggested with SCR disabled. Average Latency is around 9.8 ms, a fraction lesser. Thanks Ramu On Wed, Oct 9, 2013 at 3:32 PM, Ramu M S ramu.ma...@gmail.com wrote: Hi All, I just ran only 8 parallel clients, With SCR Enabled : Average Latency is 80 ms, IO Wait % is around 8 With SCR Disabled: Average Latency is 40 ms, IO Wait % is around 2 I always thought SCR enabled, allows a client co-located with the DataNode to read HDFS file blocks directly. This gives a performance boost to distributed clients that are aware of locality. Is my understanding wrong OR it doesn't apply to my scenario? Meanwhile I will try setting the parameter suggested by Lars and post you the results. Thanks, Ramu On Wed, Oct 9, 2013 at 2:29 PM, lars hofhansl la...@apache.org wrote: Good call. Could try to enable hbase.regionserver.checksum.verify, which will cause HBase to do its own checksums rather than relying on HDFS (and which saves 1 IO per block get). I do think you can expect the index blocks to be cached at all times. -- Lars From: Vladimir Rodionov vrodio...@carrieriq.com To: user@hbase.apache.org user@hbase.apache.org Sent: Tuesday, October 8, 2013 8:44 PM Subject: RE: HBase Random Read latency 100ms Upd. Each HBase Get = 2 HDFS read IO (index block + data block)= 4 File IO (data + .crc) in a worst case. I think if Bloom Filter is enabled than it is going to be 6 File IO in a worst case (large data set), therefore you will have not 5 IO requests in queue but up to 20-30 IO requests in a queue This definitely explains 100ms avg latency. Best regards, Vladimir Rodionov Principal Platform Engineer Carrier IQ, www.carrieriq.com e-mail: vrodio...@carrieriq.com From: Vladimir Rodionov Sent: Tuesday, October 08, 2013 7:24 PM To: user@hbase.apache.org Subject: RE: HBase Random Read latency 100ms Ramu, You have 8 server boxes and 10 client. You have 40 requests in parallel - 5 per RS/DN? You have 5 requests on random reads in a IO queue of your single RAID1. With avg read latency of 10 ms, 5 requests in queue will give us 30ms. Add some overhead of HDFS + HBase and you will probably have your issue explained ? Your bottleneck is your disk system, I think. When you serve most of requests from disks as in your large data set scenario, make sure you have adequate disk sub-system and that it is configured properly. Block Cache and OS page can not help you in this case as working data set is larger than both caches. Good performance numbers in small data set scenario are explained by the fact that data fits into OS page cache and Block Cache - you do not read data from disk even if you disable block cache. Best regards, Vladimir Rodionov Principal Platform Engineer Carrier IQ, www.carrieriq.com e-mail: vrodio...@carrieriq.com From: Ramu M S [ramu.ma...@gmail.com] Sent: Tuesday, October 08, 2013 6:00 PM To: user@hbase.apache.org Subject: Re: HBase Random Read latency 100ms Hi All, After few suggestions from the mails earlier I changed the following, 1. Heap Size to 16 GB 2. Block Size to 16KB 3. HFile size to 8 GB (Table now has 256 regions, 32 per server) 4. Data Locality Index is 100 in all RS I have clients running in 10 machines, each with 4 threads. So total 40. This is same in all tests. Result: 1. Average latency is still 100ms.
Re: Spatial data posting in HBase
The point is that there are options (multiple different hammers) if HBase support for geospatial is not there or doesn't meet OP's needs. Otis -- Solr ElasticSearch Support -- http://sematext.com/ Performance Monitoring -- http://sematext.com/spm On Wed, Oct 9, 2013 at 11:14 AM, Michael Segel msegel_had...@hotmail.com wrote: And Solr has what to do with storing data in HBase? I guess its true… if all you have is a hammer… The point I was raising was that geohash isn't the most efficient way to go when you look at the problem at a global level… On Oct 9, 2013, at 9:34 AM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Consider using Solr, which provides a lot of geospatial search support. Otis Solr ElasticSearch Support http://sematext.com/ On Sep 24, 2013 8:29 AM, cto ankur...@tcs.com wrote: Hi , I am very new in HBase. Could you please let me know , how to insert spatial data (Latitude / Longitude) in HBase using Java . -- View this message in context: http://apache-hbase.679495.n3.nabble.com/Spatial-data-posting-in-HBase-tp4051123.html Sent from the HBase User mailing list archive at Nabble.com.
Hbase export failing in 0.90
Hi, I'm trying to export 1 day data from HBase (version 0.90.4-cdh3u3), but it fails every time with 'org.apache.hadoop.hbase.regionserver.LeaseException' error [1] I increased the lease value, now it just take longer to fail. Command that I used is: sudo -u hbase hbase org.apache.hadoop.hbase.mapreduce.Export tsdb /datatsdb/09-03 1 137819160 137819460 The export command starts ~7000 jobs, map mappers per node is set to 5. First ~200 task completes within 1-2 mins (each taking avg 6 sec) but after that maps don't finish and eventually get the exception. Ganglia shows increased in n/w throughout the period the MR job is running. That suggests the map tasks are working, may be taking longer to finish. I've changed the lease settings (both on master and region servers): hbase.regionserver.lease.period = 9000 hbase.rpc.timeout = 9000 hbase.client.scanner.caching = 10 Even with this high lease period (1500 Minutes), I'm getting LeaseExeptions in 15 mins [2] Any suggestion how I can resolve this issue ? Thank you! [1] 13/10/09 14:14:35 INFO mapred.JobClient: Task Id : attempt_201310091100_0004_m_000540_2, Status : FAILED org.apache.hadoop.hbase.regionserver.LeaseException: org.apache.hadoop.hbase.regionserver.LeaseException: lease '2777638348379668022' does not exist at org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:230) at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1879) at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) [2] 2013-10-09 13:14:07,341 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server data01.xx.xx.net/xx.xx.xx.xx:2181, sessionid = 0x41825a012e95be, negotiated timeout = 4 2013-10-09 13:29:09,141 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1 2013-10-09 13:29:09,144 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hbase (auth:SIMPLE) cause:org.apache.hadoop.hbase.regionserver.LeaseException: org.apache.hadoop.hbase.regionserver.LeaseException: lease '-5530645372503000173' does not exist at org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:230) at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1879) at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
Re: HBase Random Read latency 100ms
Hi Ramu, I think I saw mentions of this possibly being a GC issue though now it seems it may be a disk IO issue? 3 things: 1) http://blog.sematext.com/2013/06/24/g1-cms-java-garbage-collector/ - our G1 experience, with HBase specifically 2) If you can share some of your performance graphs (GC, disk IO, JVM memory pools, HBase specific ones, etc.) people will likely be able to provide better help 3) You can do 2) with SPM (see sig), and actually you can send email to this ML with your graphs directly from SPM. :) Otis -- Solr ElasticSearch Support -- http://sematext.com/ Performance Monitoring -- http://sematext.com/spm On Wed, Oct 9, 2013 at 3:11 AM, Ramu M S ramu.ma...@gmail.com wrote: Hi All, Sorry. There was some mistake in the tests (Clients were not reduced, forgot to change the parameter before running tests). With 8 Clients and, SCR Enabled : Average Latency is 25 ms, IO Wait % is around 8 SCR Disabled: Average Latency is 10 ms, IO Wait % is around 2 Still, SCR disabled gives better results, which confuse me. Can anyone clarify? Also, I tried setting the parameter (hbase.regionserver.checksum.verify as true) Lars suggested with SCR disabled. Average Latency is around 9.8 ms, a fraction lesser. Thanks Ramu On Wed, Oct 9, 2013 at 3:32 PM, Ramu M S ramu.ma...@gmail.com wrote: Hi All, I just ran only 8 parallel clients, With SCR Enabled : Average Latency is 80 ms, IO Wait % is around 8 With SCR Disabled: Average Latency is 40 ms, IO Wait % is around 2 I always thought SCR enabled, allows a client co-located with the DataNode to read HDFS file blocks directly. This gives a performance boost to distributed clients that are aware of locality. Is my understanding wrong OR it doesn't apply to my scenario? Meanwhile I will try setting the parameter suggested by Lars and post you the results. Thanks, Ramu On Wed, Oct 9, 2013 at 2:29 PM, lars hofhansl la...@apache.org wrote: Good call. Could try to enable hbase.regionserver.checksum.verify, which will cause HBase to do its own checksums rather than relying on HDFS (and which saves 1 IO per block get). I do think you can expect the index blocks to be cached at all times. -- Lars From: Vladimir Rodionov vrodio...@carrieriq.com To: user@hbase.apache.org user@hbase.apache.org Sent: Tuesday, October 8, 2013 8:44 PM Subject: RE: HBase Random Read latency 100ms Upd. Each HBase Get = 2 HDFS read IO (index block + data block)= 4 File IO (data + .crc) in a worst case. I think if Bloom Filter is enabled than it is going to be 6 File IO in a worst case (large data set), therefore you will have not 5 IO requests in queue but up to 20-30 IO requests in a queue This definitely explains 100ms avg latency. Best regards, Vladimir Rodionov Principal Platform Engineer Carrier IQ, www.carrieriq.com e-mail: vrodio...@carrieriq.com From: Vladimir Rodionov Sent: Tuesday, October 08, 2013 7:24 PM To: user@hbase.apache.org Subject: RE: HBase Random Read latency 100ms Ramu, You have 8 server boxes and 10 client. You have 40 requests in parallel - 5 per RS/DN? You have 5 requests on random reads in a IO queue of your single RAID1. With avg read latency of 10 ms, 5 requests in queue will give us 30ms. Add some overhead of HDFS + HBase and you will probably have your issue explained ? Your bottleneck is your disk system, I think. When you serve most of requests from disks as in your large data set scenario, make sure you have adequate disk sub-system and that it is configured properly. Block Cache and OS page can not help you in this case as working data set is larger than both caches. Good performance numbers in small data set scenario are explained by the fact that data fits into OS page cache and Block Cache - you do not read data from disk even if you disable block cache. Best regards, Vladimir Rodionov Principal Platform Engineer Carrier IQ, www.carrieriq.com e-mail: vrodio...@carrieriq.com From: Ramu M S [ramu.ma...@gmail.com] Sent: Tuesday, October 08, 2013 6:00 PM To: user@hbase.apache.org Subject: Re: HBase Random Read latency 100ms Hi All, After few suggestions from the mails earlier I changed the following, 1. Heap Size to 16 GB 2. Block Size to 16KB 3. HFile size to 8 GB (Table now has 256 regions, 32 per server) 4. Data Locality Index is 100 in all RS I have clients running in 10 machines, each with 4 threads. So total 40. This is same in all tests. Result: 1. Average latency is still 100ms. 2. Heap occupancy is around 2-2.5 GB in all RS Few more tests carried out yesterday, TEST 1: Small data set (100 Million records, each with 724 bytes). === Configurations: 1. Heap Size to 1 GB 2. Block Size to 16KB 3. HFile size to 1 GB (Table now has 128
Coprocessor Increments
Hi All, We have been running into an RPC deadlock issue on HBase and from investigation, we believe the root of the issue is in us doing cross region increments from a coprocessor. After some further searching and reading over this http://mail-archives.apache.org/mod_mbox/hbase-user/201212.mbox/%3CCA+RK=_BP8k1Z-gQ+38RiipKgzi+=5cn3ekzdjz_z-2qt8xo...@mail.gmail.com%3E we think that we can solve this by doing the increments locally on the region. My question, is what happens if the row value specified does not land in the current region. We can obviously do our best to make sure that it does, but is there any way to be absolutely sure that it is? This is supposing we use incrementColumnValue() out of the HRegion class ( http://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/regionserver/HRegion.html#incrementColumnValue(byte[], byte[], byte[], long, boolean) http://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/regionserver/HRegion.html#incrementColumnValue%28byte[],%20byte[],%20byte[],%20long,%20boolean%29) Here is the method signature for simplicity public long*incrementColumnValue*(byte[] row, byte[] family, byte[] qualifier, long amount, boolean writeToWAL) If we specify a row, there seems to be no guarantee that row will be confined to the region the coprocessor is on. If the call does force the increment to be on the same region, what will happen if a later call ends up on another region but with the same name. Contrived Example Insert rowkey California-12345 triggers a coprocessor to call incrementColumnValue() with a rowkey of California-total all on Region 1. This would likely be on an insert on the same region. But as the table grows, this secondary insert could end up on another region. If it is confined, then suppose we later insert California-95424 which still triggers a call to incrementColumnValue() with a rowkey of California-total all on Region 2. Are we now left with two rowkeys of California-total? One on each region server? If so, what happens if these two regions are compacted into one? Hopefully this all makes sense. We are on Hbase 0.94.10. If we are going about this all wrong, that could be the issue as well :) Thanks. -John Weatherford
RE: Coprocessor Increments
Contrived Example Insert rowkey California-12345 triggers a coprocessor to call incrementColumnValue() with a rowkey of California-total all on Region 1. This would likely be on an insert on the same region. But as the table grows, this secondary insert could end up on another region. If it is confined, then suppose we later insert California-95424 which still triggers a call to incrementColumnValue() with a rowkey of California-total all on Region 2. Are we now left with two rowkeys of California-total? One on each region server? If so, what happens if these two regions are compacted into one? Best regards, Nope, Your California-total will migrate to Region 2 after region split is complete. Vladimir Rodionov Principal Platform Engineer Carrier IQ, www.carrieriq.com e-mail: vrodio...@carrieriq.com Confidentiality Notice: The information contained in this message, including any attachments hereto, may be confidential and is intended to be read only by the individual or entity to whom this message is addressed. If the reader of this message is not the intended recipient or an agent or designee of the intended recipient, please note that any review, use, disclosure or distribution of this message or its attachments, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or notificati...@carrieriq.com and delete or destroy any copy of this message and its attachments.
Re: Coprocessor Increments
John: Looks like you need a 'California-total' row in each region. Custom Split Policy can help you achieve that, see 9.7.4.1 in: http://hbase.apache.org/book.html#d0e6541 Cheers On Wed, Oct 9, 2013 at 7:28 PM, Vladimir Rodionov vrodio...@carrieriq.comwrote: Contrived Example Insert rowkey California-12345 triggers a coprocessor to call incrementColumnValue() with a rowkey of California-total all on Region 1. This would likely be on an insert on the same region. But as the table grows, this secondary insert could end up on another region. If it is confined, then suppose we later insert California-95424 which still triggers a call to incrementColumnValue() with a rowkey of California-total all on Region 2. Are we now left with two rowkeys of California-total? One on each region server? If so, what happens if these two regions are compacted into one? Best regards, Nope, Your California-total will migrate to Region 2 after region split is complete. Vladimir Rodionov Principal Platform Engineer Carrier IQ, www.carrieriq.com e-mail: vrodio...@carrieriq.com Confidentiality Notice: The information contained in this message, including any attachments hereto, may be confidential and is intended to be read only by the individual or entity to whom this message is addressed. If the reader of this message is not the intended recipient or an agent or designee of the intended recipient, please note that any review, use, disclosure or distribution of this message or its attachments, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or notificati...@carrieriq.com and delete or destroy any copy of this message and its attachments.
Re: Coprocessor Increments
bq. 'California-total' row in each region Of course such row key needs to be suffixed with either start or end key of the corresponding region. On Wed, Oct 9, 2013 at 7:39 PM, Ted Yu yuzhih...@gmail.com wrote: John: Looks like you need a 'California-total' row in each region. Custom Split Policy can help you achieve that, see 9.7.4.1 in: http://hbase.apache.org/book.html#d0e6541 Cheers On Wed, Oct 9, 2013 at 7:28 PM, Vladimir Rodionov vrodio...@carrieriq.com wrote: Contrived Example Insert rowkey California-12345 triggers a coprocessor to call incrementColumnValue() with a rowkey of California-total all on Region 1. This would likely be on an insert on the same region. But as the table grows, this secondary insert could end up on another region. If it is confined, then suppose we later insert California-95424 which still triggers a call to incrementColumnValue() with a rowkey of California-total all on Region 2. Are we now left with two rowkeys of California-total? One on each region server? If so, what happens if these two regions are compacted into one? Best regards, Nope, Your California-total will migrate to Region 2 after region split is complete. Vladimir Rodionov Principal Platform Engineer Carrier IQ, www.carrieriq.com e-mail: vrodio...@carrieriq.com Confidentiality Notice: The information contained in this message, including any attachments hereto, may be confidential and is intended to be read only by the individual or entity to whom this message is addressed. If the reader of this message is not the intended recipient or an agent or designee of the intended recipient, please note that any review, use, disclosure or distribution of this message or its attachments, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or notificati...@carrieriq.com and delete or destroy any copy of this message and its attachments.
Re: Coprocessor Increments
First, Thank you everyone for the quick response. Ted: The custom split policy could be an interesting solution. Regarding the other option you sent, if I pull the end row key, I could construct an end key, but if I suffix the row key with the end key of the region, would that actually solve my problem? In the contrived case, wouldn't the suffixed key now be California-total-California-12346 and the other region's counter would be California-total-California-95424 and both of these keys would actually end up on the second region since the sorting of the keys would impose that California-t* comes after California-[1-9]. Part of the question is that I don't understand if calling incrementColumnValue() on the region will always execute on the region called, regardles of rowkey. If so, what happens when those regions are merged? Thanks again for the help! On Wed 09 Oct 2013 07:43:34 PM PDT, Ted Yu wrote: bq. 'California-total' row in each region Of course such row key needs to be suffixed with either start or end key of the corresponding region. On Wed, Oct 9, 2013 at 7:39 PM, Ted Yu yuzhih...@gmail.com wrote: John: Looks like you need a 'California-total' row in each region. Custom Split Policy can help you achieve that, see 9.7.4.1 in: http://hbase.apache.org/book.html#d0e6541 Cheers On Wed, Oct 9, 2013 at 7:28 PM, Vladimir Rodionov vrodio...@carrieriq.com wrote: Contrived Example Insert rowkey California-12345 triggers a coprocessor to call incrementColumnValue() with a rowkey of California-total all on Region 1. This would likely be on an insert on the same region. But as the table grows, this secondary insert could end up on another region. If it is confined, then suppose we later insert California-95424 which still triggers a call to incrementColumnValue() with a rowkey of California-total all on Region 2. Are we now left with two rowkeys of California-total? One on each region server? If so, what happens if these two regions are compacted into one? Best regards, Nope, Your California-total will migrate to Region 2 after region split is complete. Vladimir Rodionov Principal Platform Engineer Carrier IQ, www.carrieriq.com e-mail: vrodio...@carrieriq.com Confidentiality Notice: The information contained in this message, including any attachments hereto, may be confidential and is intended to be read only by the individual or entity to whom this message is addressed. If the reader of this message is not the intended recipient or an agent or designee of the intended recipient, please note that any review, use, disclosure or distribution of this message or its attachments, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or notificati...@carrieriq.com and delete or destroy any copy of this message and its attachments.
Re: Coprocessor Increments
John:
Suppose 'California-**12346' is the start key of region 1 and 'California-**
95424' is the start key of region 2. You can choose 'California-**12346#total'
to be the row key where increment is done for region 1 and
'California-**95424#total'
to be the row key where increment is done for region 2.
w.r.t. verification of whether given row key is indeed inside the hosting
region, see the following:
void checkRow(final byte [] row, String op) throws IOException {
if (!rowIsInRange(getRegionInfo(), row)) {
throw new WrongRegionException(Requested row out of range for +
op + on HRegion + this + , startKey=' +
Cheers
On Wed, Oct 9, 2013 at 8:26 PM, John Weatherford
john.weatherf...@telescope.tv wrote:
First, Thank you everyone for the quick response.
Ted:
The custom split policy could be an interesting solution. Regarding the
other option you sent, if I pull the end row key, I could construct an end
key, but if I suffix the row key with the end key of the region, would that
actually solve my problem? In the contrived case, wouldn't the suffixed key
now be California-total-California-**12346 and the other region's
counter would be California-total-California-**95424 and both of these
keys would actually end up on the second region since the sorting of the
keys would impose that California-t* comes after California-[1-9].
Part of the question is that I don't understand if calling
incrementColumnValue() on the region will always execute on the region
called, regardles of rowkey. If so, what happens when those regions are
merged?
Thanks again for the help!
On Wed 09 Oct 2013 07:43:34 PM PDT, Ted Yu wrote:
bq. 'California-total' row in each region
Of course such row key needs to be suffixed with either start or end key
of
the corresponding region.
On Wed, Oct 9, 2013 at 7:39 PM, Ted Yu yuzhih...@gmail.com wrote:
John:
Looks like you need a 'California-total' row in each region.
Custom Split Policy can help you achieve that, see 9.7.4.1 in:
http://hbase.apache.org/book.**html#d0e6541http://hbase.apache.org/book.html#d0e6541
Cheers
On Wed, Oct 9, 2013 at 7:28 PM, Vladimir Rodionov
vrodio...@carrieriq.com
wrote:
Contrived Example
Insert rowkey California-12345 triggers a coprocessor to call
incrementColumnValue() with a rowkey of California-total all on
Region 1.
This would likely be on an insert on the same region. But as the table
grows, this secondary insert could end up on another region. If it is
confined, then suppose we later insert California-95424 which still
triggers a call to incrementColumnValue() with a rowkey of
California-total all on Region 2.
Are we now left with two rowkeys of California-total? One on each
region server? If so, what happens if these two regions are compacted
into one?
Best regards,
Nope, Your California-total will migrate to Region 2 after region
split
is complete.
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com
__**__
Confidentiality Notice: The information contained in this message,
including any attachments hereto, may be confidential and is intended
to be
read only by the individual or entity to whom this message is
addressed. If
the reader of this message is not the intended recipient or an agent or
designee of the intended recipient, please note that any review, use,
disclosure or distribution of this message or its attachments, in any
form,
is strictly prohibited. If you have received this message in error,
please
immediately notify the sender and/or notificati...@carrieriq.com and
delete or destroy any copy of this message and its attachments.
Re: Coprocessor Increments
Thank you ted. I might have to rethink my key design to accomodate
this. With this design and using the totals in keys as you suggested, I
would have to scan the entire California group to find the totals.
Thank you very much for your help.
-John
On Wed 09 Oct 2013 08:43:12 PM PDT, Ted Yu wrote:
John:
Suppose 'California-**12346' is the start key of region 1 and 'California-**
95424' is the start key of region 2. You can choose 'California-**12346#total'
to be the row key where increment is done for region 1 and
'California-**95424#total'
to be the row key where increment is done for region 2.
w.r.t. verification of whether given row key is indeed inside the hosting
region, see the following:
void checkRow(final byte [] row, String op) throws IOException {
if (!rowIsInRange(getRegionInfo(), row)) {
throw new WrongRegionException(Requested row out of range for +
op + on HRegion + this + , startKey=' +
Cheers
On Wed, Oct 9, 2013 at 8:26 PM, John Weatherford
john.weatherf...@telescope.tv wrote:
First, Thank you everyone for the quick response.
Ted:
The custom split policy could be an interesting solution. Regarding the
other option you sent, if I pull the end row key, I could construct an end
key, but if I suffix the row key with the end key of the region, would that
actually solve my problem? In the contrived case, wouldn't the suffixed key
now be California-total-California-**12346 and the other region's
counter would be California-total-California-**95424 and both of these
keys would actually end up on the second region since the sorting of the
keys would impose that California-t* comes after California-[1-9].
Part of the question is that I don't understand if calling
incrementColumnValue() on the region will always execute on the region
called, regardles of rowkey. If so, what happens when those regions are
merged?
Thanks again for the help!
On Wed 09 Oct 2013 07:43:34 PM PDT, Ted Yu wrote:
bq. 'California-total' row in each region
Of course such row key needs to be suffixed with either start or end key
of
the corresponding region.
On Wed, Oct 9, 2013 at 7:39 PM, Ted Yu yuzhih...@gmail.com wrote:
John:
Looks like you need a 'California-total' row in each region.
Custom Split Policy can help you achieve that, see 9.7.4.1 in:
http://hbase.apache.org/book.**html#d0e6541http://hbase.apache.org/book.html#d0e6541
Cheers
On Wed, Oct 9, 2013 at 7:28 PM, Vladimir Rodionov
vrodio...@carrieriq.com
wrote:
Contrived Example
Insert rowkey California-12345 triggers a coprocessor to call
incrementColumnValue() with a rowkey of California-total all on
Region 1.
This would likely be on an insert on the same region. But as the table
grows, this secondary insert could end up on another region. If it is
confined, then suppose we later insert California-95424 which still
triggers a call to incrementColumnValue() with a rowkey of
California-total all on Region 2.
Are we now left with two rowkeys of California-total? One on each
region server? If so, what happens if these two regions are compacted
into one?
Best regards,
Nope, Your California-total will migrate to Region 2 after region
split
is complete.
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodio...@carrieriq.com
__**__
Confidentiality Notice: The information contained in this message,
including any attachments hereto, may be confidential and is intended
to be
read only by the individual or entity to whom this message is
addressed. If
the reader of this message is not the intended recipient or an agent or
designee of the intended recipient, please note that any review, use,
disclosure or distribution of this message or its attachments, in any
form,
is strictly prohibited. If you have received this message in error,
please
immediately notify the sender and/or notificati...@carrieriq.com and
delete or destroy any copy of this message and its attachments.
