Thanks for the info, Dimitri, but I don't see anything Phoenix-specific in that stack trace. Are you using immutable indexes (i.e. declaring IMMUTABLE_ROWS=true on your CREATE TABLE statement), or are you using mutable indexes?
Mujtaba - any ideas? How long should it take? Do you think Dimitri should update any of the default config params? Jesse - any ideas - does this stack track look suspicious to you? Thanks, James On Thu, Jan 30, 2014 at 12:16 PM, Wijesinghe, Dimitri < [email protected]> wrote: > Hi James, > > Thanks for your reply. Updated to 2.2.2 and the issue persisted. I went > into the master node's regionserver log and found this exception (copied > the entire section that mentioned the index, which I had called idx) > > hduser@C-Master:/usr/local/hadoop/hbase/logs$ vim > hbase-hduser-regionserver-C-Ma > ster.out > 14/01/30 19:52:37 INFO regionserver.HRegion: Starting compaction on _0 in > region > IDX,\x09\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1391110815630.18468269b48eec607d41e5e52fa59ffe. > 14/01/30 19:52:37 INFO regionserver.Store: Starting compaction of 4 > file(s) in _0 of > IDX,\x09\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1391110815630.18468269b48eec607d41e5e52fa59ffe. > into > tmpdir=hdfs://C-Master:54310/hbase/IDX/18468269b48eec607d41e5e52fa59ffe/.tmp, > seqid=375945, totalSize=65.6m > 14/01/30 19:52:37 INFO util.FSUtils: FileSystem doesn't support > getDefaultReplication > 14/01/30 19:52:37 INFO util.FSUtils: FileSystem doesn't support > getDefaultBlockSize > 14/01/30 19:52:37 INFO regionserver.StoreFile: Delete Family Bloom filter > type for > hdfs://C-Master:54310/hbase/IDX/18468269b48eec607d41e5e52fa59ffe/.tmp/97cf82bd088a4b03ae0d169cfeddecdf: > CompoundBloomFilterWriter > 14/01/30 19:52:49 INFO regionserver.StoreFile: NO General Bloom and NO > DeleteFamily was added to HFile > (hdfs://C-Master:54310/hbase/IDX/18468269b48eec607d41e5e52fa59ffe/.tmp/97cf82bd088a4b03ae0d169cfeddecdf) > 14/01/30 19:52:49 INFO regionserver.Store: Renaming compacted file at > hdfs://C-Master:54310/hbase/IDX/18468269b48eec607d41e5e52fa59ffe/.tmp/97cf82bd088a4b03ae0d169cfeddecdf > to > hdfs://C-Master:54310/hbase/IDX/18468269b48eec607d41e5e52fa59ffe/_0/97cf82bd088a4b03ae0d169cfeddecdf > 14/01/30 19:52:50 INFO regionserver.Store: Completed major compaction of 4 > file(s) in _0 of > IDX,\x09\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1391110815630.18468269b48eec607d41e5e52fa59ffe. > into 97cf82bd088a4b03ae0d169cfeddecdf, size=63.5m; total size for store is > 63.5m > 14/01/30 19:52:50 INFO compactions.CompactionRequest: completed > compaction: > regionName=IDX,\x09\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1391110815630.18468269b48eec607d41e5e52fa59ffe., > storeName=_0, fileCount=4, fileSize=65.6m, priority=3, time=1151466089681; > duration=12sec > 14/01/30 19:55:19 WARN ipc.HBaseServer: IPC Server listener on 60020: > readAndProcess threw exception java.io.IOException: Connection reset by > peer. Count of bytes read: 0 > java.io.IOException: Connection reset by peer > at sun.nio.ch.FileDispatcherImpl.read0(Native Method) > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:225) > at sun.nio.ch.IOUtil.read(IOUtil.java:198) > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359) > at > org.apache.hadoop.hbase.ipc.HBaseServer.channelRead(HBaseServer.java:1796) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:1179) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:748) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:539) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:722) > > Does this shed any light on the issue? > > Thanks, > -D > > > On Thu, Jan 30, 2014 at 12:35 PM, James Taylor <[email protected]>wrote: > >> Hi Dimitri, >> Please try with 2.2.2 and let us know if the problem occurs. If yes, then >> check your server logs for any exceptions and send that along as well. >> Thanks, >> James >> >> >> On Thursday, January 30, 2014, Wijesinghe, Dimitri < >> [email protected]> wrote: >> >>> Hi, >>> >>> I am trying to create an index for a table with about 100M records, with >>> 10 SALT buckets across a five-node cluster. Running HBase 0.94.12 and >>> Phoenix 2.1.2. >>> >>> When I issue the following command in sqlline.sh I get the error message >>> below after a roughly 10-minute wait. >>> >>> 0: jdbc:phoenix:C-Master,C-Slave,C-Slave2,C-S> CREATE INDEX idx ON >>> WageRate(HospID,CostCenter,JobCode,Rate,PrimaryKey); >>> >>> Error: (state=08000,code=101) >>> >>> I believe this is just the general Phoenix error code so I am not what >>> the exact problem is. I had previously tried this operation on a 1M record >>> table (the 100M table is basically a 100 copies of the previous table with >>> different primary keys). The indexing on the 1M table worked perfectly. >>> >>> Any help would be greatly appreciated as un-indexed operations on the >>> cluster are now taking a fairly significant amount of time. >>> >>> Thanks! >>> -D >>> >>> -- >>> *Dimitri Wijesinghe* >>> >>> *iVantage Health Analytics*® >>> >>> Web Developer >>> >>> 300 Chestnut Street, Suite 101 | Needham, MA 02492 >>> >>> direct: 781-247-20*91* | office: 781-499-5287 >>> >>> email: [email protected] | >>> www.iVantageHealth.com<http://www.ivantagehealth.com/> >>> >> > > > -- > *Dimitri Wijesinghe* > > *iVantage Health Analytics*® > > Web Developer > > 300 Chestnut Street, Suite 101 | Needham, MA 02492 > > direct: 781-247-20*91* | office: 781-499-5287 > > email: [email protected] | > www.iVantageHealth.com<http://www.ivantagehealth.com/> >
