> > With the new stats feature in 3.2/4.2, salting tables is less > necessary and will likely decrease your overall cluster throughput.
Interesting, where can I get details, James? Is it fast region reassignment based on load statistics? -Vladimir Rodionov On Fri, Nov 7, 2014 at 10:29 AM, James Taylor <[email protected]> wrote: > If you salt your table (which pre-splits the table into SALT_BUCKETS > regions), by default your index will be salted and pre-split the same > way. > > FWIW, you can also presplit your table and index using the SPLIT ON > (...) syntax: http://phoenix.apache.org/language/index.html#create_table > > With the new stats feature in 3.2/4.2, salting tables is less > necessary and will likely decrease your overall cluster throughput. > > Thanks, > James > > > > On Fri, Nov 7, 2014 at 10:21 AM, Vladimir Rodionov > <[email protected]> wrote: > > If you see split activity on your index tables, they are either not pre > > split or region sizes exceeds max limit (you load a lot of data into > > indexes) or index tables still on default split policy. > > > > How do you pre split your index tables? > > > > -Vladimir Rodionov > > > > On Fri, Nov 7, 2014 at 7:40 AM, Perko, Ralph J <[email protected]> > wrote: > >> > >> Salting the table, (gives me pre-splits), and using the split policy of > >> ConstantSizeRegionSplitPolicy as you suggested worked! > >> > >> A question on the index tables – unlike the main table, hbase shows that > >> each index table has the MAX_FILESIZE attribute set to 344148020 bytes > which > >> is well below what is set for the max HStoreFile size property in hbase, > >> causing a large amount of splitting on all the index tables (which are > pre > >> split as well) despite using the same split policy as the main table. > Why > >> is this done for just the index tables? Is it safe to override? > >> > >> Thanks, > >> Ralph > >> __________________________________________________ > >> Ralph Perko > >> Pacific Northwest National Laboratory > >> (509) 375-2272 > >> [email protected] > >> > >> > >> From: Vladimir Rodionov <[email protected]> > >> Reply-To: "[email protected]" <[email protected]> > >> Date: Thursday, November 6, 2014 at 1:04 PM > >> To: "[email protected]" <[email protected]> > >> Subject: Re: RegionTooBusyException > >> > >> You may want to try different RegionSplitPolicy > >> (ConstantSizeRegionSplitPolicy), default one > >> (IncreasingToUpperBoundRegionSplitPolicy) > >> does not make sense when table is prespit in advance. > >> > >> -Vladimir Rodionov > >> > >> On Thu, Nov 6, 2014 at 1:01 PM, Vladimir Rodionov < > [email protected]> > >> wrote: > >>> > >>> Too many map task trying concurrently commit (save) data to HBase, I > bet > >>> you have compaction hell in your cluster during data loading. > >>> > >>> In a few words, you cluster is not able to keep up with data ingestion > >>> rate. HBase does not do smart update/insert rate throttling for you. > You may > >>> try some compaction - related configuration options : > >>> hbase.hstore.blockingWaitTime - Default. 90000 > >>> hbase.hstore.compaction.min - Default. 3 > >>> hbase.hstore.compaction.max - Default. 10 > >>> hbase.hstore.compaction.min.size - Default: 128 MB expressed in > bytes. > >>> > >>> but I suggest you to pre-split your tables first, than limit # of map > >>> tasks (if former does not help), than play with compaction config > values > >>> (above). > >>> > >>> > >>> -Vladimir Rodionov > >>> > >>> On Thu, Nov 6, 2014 at 12:31 PM, Perko, Ralph J <[email protected]> > >>> wrote: > >>>> > >>>> Hi, I am using a combination of Pig, Phoenix and HBase to load data > on a > >>>> test cluster and I continue to run into an issue with larger, longer > running > >>>> jobs (smaller jobs succeed). After the job has run for several > hours, the > >>>> first set of mappers have finished and the second begin, the job dies > with > >>>> each mapper failing with the error RegionTooBusyException. Could > this be > >>>> related to how I have my Phoenix tables configured or is this an Hbase > >>>> configuration issue or something else? Do you have any suggestions? > >>>> > >>>> Thanks for the help, > >>>> Ralph > >>>> > >>>> > >>>> 2014-11-05 23:08:31,573 INFO [main] > >>>> org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 200 > actions to > >>>> finish > >>>> 2014-11-05 23:08:33,729 WARN [phoenix-1-thread-34413] > >>>> org.apache.hadoop.hbase.client.AsyncProcess: #1, table=T1_CSV_DATA, > primary, > >>>> attempt=36/35 failed 200 ops, last exception: null on > >>>> server1,60020,1415229553858, tracking started Wed Nov 05 22:59:40 PST > 2014; > >>>> not retrying 200 - final failure > >>>> 2014-11-05 23:08:33,736 WARN [main] > org.apache.hadoop.mapred.YarnChild: > >>>> Exception running child : java.io.IOException: Exception while > committing to > >>>> database. > >>>> at > >>>> > org.apache.phoenix.pig.hadoop.PhoenixRecordWriter.write(PhoenixRecordWriter.java:79) > >>>> at > >>>> > org.apache.phoenix.pig.hadoop.PhoenixRecordWriter.write(PhoenixRecordWriter.java:41) > >>>> at > >>>> > org.apache.phoenix.pig.PhoenixHBaseStorage.putNext(PhoenixHBaseStorage.java:151) > >>>> at > >>>> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:139) > >>>> at > >>>> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:98) > >>>> at > >>>> > org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:635) > >>>> at > >>>> > org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89) > >>>> at > >>>> > org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112) > >>>> at > >>>> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.collect(PigMapOnly.java:48) > >>>> at > >>>> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:284) > >>>> at > >>>> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:277) > >>>> at > >>>> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64) > >>>> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) > >>>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) > >>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) > >>>> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) > >>>> at java.security.AccessController.doPrivileged(Native Method) > >>>> at javax.security.auth.Subject.doAs(Subject.java:396) > >>>> at > >>>> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594) > >>>> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) > >>>> Caused by: org.apache.phoenix.execute.CommitException: > >>>> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: > Failed > >>>> 200 actions: RegionTooBusyException: 200 times, > >>>> at > >>>> > org.apache.phoenix.execute.MutationState.commit(MutationState.java:418) > >>>> at > >>>> > org.apache.phoenix.jdbc.PhoenixConnection.commit(PhoenixConnection.java:356) > >>>> at > >>>> > org.apache.phoenix.pig.hadoop.PhoenixRecordWriter.write(PhoenixRecordWriter.java:76) > >>>> ... 19 more > >>>> Caused by: > >>>> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: > Failed > >>>> 200 actions: RegionTooBusyException: 200 times, > >>>> at > >>>> > org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:207) > >>>> at > >>>> > org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1700(AsyncProcess.java:187) > >>>> at > >>>> > org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.getErrors(AsyncProcess.java:1473) > >>>> at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:855) > >>>> at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:869) > >>>> at > >>>> > org.apache.phoenix.execute.MutationState.commit(MutationState.java:399) > >>>> ... 21 more > >>>> > >>>> 2014-11-05 23:08:33,739 INFO [main] org.apache.hadoop.mapred.Task: > >>>> Runnning cleanup for the task > >>>> 2014-11-05 23:08:33,773 INFO [Thread-11] > >>>> > org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation: > >>>> Closing zookeeper sessionid=0x2497d0ab7e6007e > >>>> > >>>> Data size: > >>>> 75 csv files compressed with bz2 > >>>> 17g compressed – 165g Uncompressed > >>>> > >>>> Time-series data, 6 node cluster, 5 region servers. Hadoop 2.5 (HDP > >>>> 2.1.5). Phoenix 4.0, Hbase 0.98, > >>>> > >>>> Phoenix Table def: > >>>> > >>>> CREATE TABLE IF NOT EXISTS > >>>> t1_csv_data > >>>> ( > >>>> timestamp BIGINT NOT NULL, > >>>> location VARCHAR NOT NULL, > >>>> fileid VARCHAR NOT NULL, > >>>> recnum INTEGER NOT NULL, > >>>> field5 VARCHAR, > >>>> ... > >>>> field45 VARCHAR, > >>>> CONSTRAINT pkey PRIMARY KEY (timestamp, > >>>> location, fileid,recnum) > >>>> ) > >>>> IMMUTABLE_ROWS=true,COMPRESSION='SNAPPY',SALT_BUCKETS=10; > >>>> > >>>> -- indexes > >>>> CREATE INDEX t1_csv_data_f1_idx ON t1_csv_data(somefield1) > >>>> COMPRESSION='SNAPPY'; > >>>> CREATE INDEX t1_csv_data_f2_idx ON t1_csv_data(somefield2) > >>>> COMPRESSION='SNAPPY'; > >>>> CREATE INDEX t1_csv_data_f3_idx ON t1_csv_data(somefield3) > >>>> COMPRESSION='SNAPPY'; > >>>> > >>>> Simple Pig script: > >>>> > >>>> register $phoenix_jar; > >>>> register $udf_jar; > >>>> Z = load '$data' as ( > >>>> file_id, > >>>> recnum, > >>>> dtm:chararray, > >>>> ... > >>>> -- lots of other fields > >>>> ); > >>>> D = foreach Z generate > >>>> gov.pnnl.pig.TimeStringToPeriod(dtm,'yyyyMMdd > >>>> HH:mm:ss','yyyyMMddHHmmss'), > >>>> location, > >>>> fileid, > >>>> recnum, > >>>> ... > >>>> -- lots of other fields > >>>> ; > >>>> STORE D into > >>>> 'hbase://$table_name' using > >>>> org.apache.phoenix.pig.PhoenixHBaseStorage('$zookeeper','-batchSize > >>>> 1000'); > >>>> > >>> > >> > > >
