Hi, I'm using Phoenix4.9.0/HBase1.3.0 with Spark2.1.1 on AWS EMR 5.6.0.
When one side uses Spark Streaming to write a lot of data and the other side uses Spark to read data, I encounter several similar exceptions. 17/07/05 04:33:47 ERROR Executor: Exception in task 83.0 in stage 166.0 (TID 95211) org.apache.ibatis.exceptions.PersistenceException: ### Error querying database. Cause: java.sql.SQLException: ERROR 201 (22000): Illegal data. ERROR 201 (22000): ERROR 201 (22000): Illegal data. Expected length of at least 8 bytes, but had 4 TRP,\x0B\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1499228698442.4f312a19bfdc624776516228c9c4f2d5. ### The error may exist in agp/mapper/TripMapper.java (best guess) ### The error may involve agp.mapper.TripMapper.selectLast ### The error occurred while handling results ### SQL: SELECT trid, tid, frtp, frno, gzid, ontm, onty, onlt, onln, oftm, ofty, oflt, ofln, onwk, onhr, wday, dist, drtn, delf, sntf, cdtm, udtm FROM trp WHERE tid = ? AND delf = FALSE ORDER BY oftm DESC NULLS LAST FETCH FIRST 1 ROW ONLY ### Cause: java.sql.SQLException: ERROR 201 (22000): Illegal data. ERROR 201 (22000): ERROR 201 (22000): Illegal data. Expected length of at least 8 bytes, but had 4 TRP,\x0B\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1499228698442.4f312a19bfdc624776516228c9c4f2d5. at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30) at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:150) at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:141) at org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne(DefaultSqlSession.java:77) at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:82) at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:59) at com.sun.proxy.$Proxy38.selectLast(Unknown Source) at agp.dao.TripDAO.selectLast(TripDAO.java:55) at agp.trip.func.trip.SelectTripCondition.fixAccOnLocation(SelectTripCondition.java:128) at agp.trip.func.trip.SelectTripCondition.call(SelectTripCondition.java:65) at agp.trip.func.trip.SelectTripCondition.call(SelectTripCondition.java:25) at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1.apply(JavaRDDLike.scala:125) at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1.apply(JavaRDDLike.scala:125) at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434) at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440) at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:461) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408) at org.apache.spark.shuffle.sort.UnsafeShuffleWriter.write(UnsafeShuffleWriter.java:166) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53) at org.apache.spark.scheduler.Task.run(Task.scala:99) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: java.sql.SQLException: ERROR 201 (22000): Illegal data. ERROR 201 (22000): ERROR 201 (22000): Illegal data. Expected length of at least 8 bytes, but had 4 TRP,\x0B\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1499228698442.4f312a19bfdc624776516228c9c4f2d5. at org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:464) at org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:150) at org.apache.phoenix.util.ServerUtil.parseRemoteException(ServerUtil.java:134) at org.apache.phoenix.util.ServerUtil.parseServerExceptionOrNull(ServerUtil.java:123) at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:109) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:755) at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:699) at org.apache.phoenix.iterate.MergeSortResultIterator.getMinHeap(MergeSortResultIterator.java:72) at org.apache.phoenix.iterate.MergeSortResultIterator.minIterator(MergeSortResultIterator.java:93) at org.apache.phoenix.iterate.MergeSortResultIterator.next(MergeSortResultIterator.java:58) at org.apache.phoenix.iterate.MergeSortTopNResultIterator.next(MergeSortTopNResultIterator.java:95) at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:778) at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.ibatis.logging.jdbc.ResultSetLogger.invoke(ResultSetLogger.java:68) at com.sun.proxy.$Proxy37.next(Unknown Source) at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleRowValuesForSimpleResultMap(DefaultResultSetHandler.java:345) at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleRowValues(DefaultResultSetHandler.java:322) at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleResultSet(DefaultResultSetHandler.java:295) at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleResultSets(DefaultResultSetHandler.java:192) at org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:64) at org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:79) at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:63) at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:324) at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156) at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:109) at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:83) at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:148) ... 23 more Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 201 (22000): ERROR 201 (22000): Illegal data. Expected length of at least 8 bytes, but had 4 TRP,\x0B\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1499228698442.4f312a19bfdc624776516228c9c4f2d5. at org.apache.phoenix.util.ServerUtil.createIOException(ServerUtil.java:81) at org.apache.phoenix.util.ServerUtil.throwIOException(ServerUtil.java:55) at org.apache.phoenix.coprocessor.ScanRegionObserver.getTopNScanner(ScanRegionObserver.java:328) at org.apache.phoenix.coprocessor.ScanRegionObserver.doPostScannerOpen(ScanRegionObserver.java:241) at org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.overrideDelegate(BaseScannerRegionObserver.java:215) at org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.nextRaw(BaseScannerRegionObserver.java:260) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2633) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2837) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34950) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) Caused by: java.sql.SQLException: ERROR 201 (22000): Illegal data. Expected length of at least 8 bytes, but had 4 at org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:464) at org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:150) at org.apache.phoenix.util.ServerUtil.parseRemoteException(ServerUtil.java:134) at org.apache.phoenix.util.ServerUtil.parseServerExceptionOrNull(ServerUtil.java:123) at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:109) at org.apache.phoenix.iterate.RegionScannerResultIterator.next(RegionScannerResultIterator.java:61) at org.apache.phoenix.iterate.OrderedResultIterator.getResultIterator(OrderedResultIterator.java:254) at org.apache.phoenix.iterate.OrderedResultIterator.next(OrderedResultIterator.java:198) at org.apache.phoenix.coprocessor.ScanRegionObserver.getTopNScanner(ScanRegionObserver.java:323) ... 10 more Caused by: java.sql.SQLException: ERROR 201 (22000): Illegal data. Expected length of at least 8 bytes, but had 4 at org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:464) at org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:150) at org.apache.phoenix.schema.KeyValueSchema.next(KeyValueSchema.java:213) at org.apache.phoenix.expression.ProjectedColumnExpression.evaluate(ProjectedColumnExpression.java:115) at org.apache.phoenix.schema.KeyValueSchema.toBytes(KeyValueSchema.java:112) at org.apache.phoenix.execute.TupleProjector.projectResults(TupleProjector.java:245) at org.apache.phoenix.coprocessor.BaseScannerRegionObserver$1.nextRaw(BaseScannerRegionObserver.java:441) at org.apache.phoenix.iterate.RegionScannerResultIterator.next(RegionScannerResultIterator.java:50) ... 13 more at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:97) at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:272) at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:64) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:360) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:334) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65) ... 3 more HBase Region Server Logs below, 1. 2017-07-05 04:33:41,181 WARN [Finalizer] memory.GlobalMemoryManager: Orphaned chunk of 104 bytes found during finalize 2017-07-05 04:33:50,929 WARN [Finalizer] memory.GlobalMemoryManager: Orphaned chunk of 104 bytes found during finalize 2. 2017-07-05 04:32:52,087 INFO [sync.2] wal.FSHLog: Slow sync cost: 104 ms, current pipeline: [DatanodeInfoWithStorage[172.31.10.147:50010,DS-289c3237-9ec2-4abc-9fe8-e114fde75483,DISK], DatanodeInfoWithStorage[172.31.8.18:50010,DS-5887519a-6424-4bf9-9c09-35d9b13b5b1d,DISK], DatanodeInfoWithStorage[172.31.10.188:50010,DS-0c7f1e59-1719-4f5d-a400-79276b4469d1,DISK]] 2017-07-05 04:34:31,791 WARN [Finalizer] memory.GlobalMemoryManager: Orphaned chunk of 104 bytes found during finalize 3. 2017-07-05 04:32:54,447 INFO [BucketCacheStatsExecutor] bucket.BucketCache: failedBlockAdditions=0, totalSize=80.00 GB, freeSize=80.00 GB, usedSize=2.87 MB, cacheSize=1.87 MB, accesses=802, hits=261, IOhitsPerSecond=0, IOTimePerHit=0.11, hitRatio=32.54%, cachingAccesses=372, cachingHits=196, cachingHitsRatio=52.69%, evictions=0, evicted=65, evictedPerRun=Infinity 2017-07-05 04:34:18,621 WARN [Finalizer] memory.GlobalMemoryManager: Orphaned chunk of 104 bytes found during finalize 4. 2017-07-05 04:33:41,653 WARN [Finalizer] memory.GlobalMemoryManager: Orphaned chunk of 104 bytes found during finalize 2017-07-05 04:33:50,601 WARN [Finalizer] memory.GlobalMemoryManager: Orphaned chunk of 104 bytes found during finalize 5. 2017-07-05 04:32:04,566 INFO [BucketCacheStatsExecutor] bucket.BucketCache: failedBlockAdditions=0, totalSize=80.00 GB, freeSize=80.00 GB, usedSize=2.82 MB, cacheSize=1.85 MB, accesses=300, hits=114, IOhitsPerSecond=0, IOTimePerHit=0.11, hitRatio=38.00%, cachingAccesses=203, cachingHits=114, cachingHitsRatio=56.16%, evictions=0, evicted=0, evictedPerRun=NaN 2017-07-05 04:33:41,167 WARN [Finalizer] memory.GlobalMemoryManager: Orphaned chunk of 104 bytes found during finalize 6. 2017-07-05 04:31:52,910 INFO [ip-172-31-11-39.ap-northeast-1.compute.internal,16020,1499227300627-index-writer--pool30-t13] regionserver.HRegion: writing data to region TRB,\x0C\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1499228749864.f0fade12881d6c99e9674639c90acad1. with WAL disabled. Data may be lost in the event of a crash. 2017-07-05 04:32:04,504 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=11.17 MB, freeSize=10.56 GB, max=10.57 GB, blockCount=2, accesses=49007, hits=49007, hitRatio=100.00%, , cachingAccesses=49007, cachingHits=49007, cachingHitsRatio=100.00%, evictions=179, evicted=0, evictedPerRun=0.0 2017-07-05 04:32:04,581 INFO [BucketCacheStatsExecutor] bucket.BucketCache: failedBlockAdditions=0, totalSize=80.00 GB, freeSize=79.99 GB, usedSize=5.59 MB, cacheSize=3.66 MB, accesses=599, hits=218, IOhitsPerSecond=0, IOTimePerHit=0.09, hitRatio=36.39%, cachingAccesses=402, cachingHits=218, cachingHitsRatio=54.23%, evictions=0, evicted=0, evictedPerRun=NaN 2017-07-05 04:33:46,020 WARN [Finalizer] memory.GlobalMemoryManager: Orphaned chunk of 104 bytes found during finalize 7. 2017-07-05 04:31:53,027 INFO [ip-172-31-12-156.ap-northeast-1.compute.internal,16020,1499227465864-index-writer--pool26-t15] regionserver.HRegion: writing data to region TRB,\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1499228749864.31b5cfcc09891813241932c46e2e6d5b. with WAL disabled. Data may be lost in the event of a crash. 2017-07-05 04:34:10,233 WARN [Finalizer] memory.GlobalMemoryManager: Orphaned chunk of 104 bytes found during finalize 2017-07-05 04:34:29,590 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=11.17 MB, freeSize=10.56 GB, max=10.57 GB, blockCount=2, accesses=27170, hits=27170, hitRatio=100.00%, , cachingAccesses=27170, cachingHits=27170, cachingHitsRatio=100.00%, evictions=179, evicted=0, evictedPerRun=0.0 2017-07-05 04:34:29,643 INFO [BucketCacheStatsExecutor] bucket.BucketCache: failedBlockAdditions=0, totalSize=80.00 GB, freeSize=80.00 GB, usedSize=2.90 MB, cacheSize=1.90 MB, accesses=295, hits=98, IOhitsPerSecond=0, IOTimePerHit=NaN, hitRatio=33.22%, cachingAccesses=200, cachingHits=98, cachingHitsRatio=49.00%, evictions=0, evicted=0, evictedPerRun=NaN 2017-07-05 04:35:13,274 WARN [Finalizer] memory.GlobalMemoryManager: Orphaned chunk of 104 bytes found during finalize 8. 2017-07-05 04:27:41,202 ERROR [RpcServer.FifoWFPBQ.default.handler=29,queue=2,port=16020] coprocessor.MetaDataEndpointImpl: getTable failed org.apache.hadoop.hbase.NotServingRegionException: SYSTEM.CATALOG,,1499216017494.3bbe2fb5b186285a9555b45c87684cc7. is closing at org.apache.hadoop.hbase.regionserver.HRegion.startRegionOperation(HRegion.java:8291) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2645) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2640) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2634) at org.apache.phoenix.coprocessor.MetaDataEndpointImpl.buildTable(MetaDataEndpointImpl.java:533) at org.apache.phoenix.coprocessor.MetaDataEndpointImpl.doGetTable(MetaDataEndpointImpl.java:2939) at org.apache.phoenix.coprocessor.MetaDataEndpointImpl.doGetTable(MetaDataEndpointImpl.java:2884) at org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTable(MetaDataEndpointImpl.java:492) at org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService.callMethod(MetaDataProtos.java:16267) at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:8095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2072) at org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2054) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168) 2017-07-05 04:31:53,017 INFO [ip-172-31-14-56.ap-northeast-1.compute.internal,16020,1499227310987-index-writer--pool43-t18] regionserver.HRegion: writing data to region TRB,\x07\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1499228749864.c93f2a12e912e9501f0801e9cb73f75c. with WAL disabled. Data may be lost in the event of a crash. 2017-07-05 04:32:04,571 INFO [BucketCacheStatsExecutor] bucket.BucketCache: failedBlockAdditions=0, totalSize=80.00 GB, freeSize=80.00 GB, usedSize=3.01 MB, cacheSize=1.97 MB, accesses=1077, hits=881, IOhitsPerSecond=0, IOTimePerHit=0.16, hitRatio=81.80%, cachingAccesses=980, cachingHits=881, cachingHitsRatio=89.90%, evictions=0, evicted=0, evictedPerRun=NaN 2017-07-05 04:36:34,114 WARN [Finalizer] memory.GlobalMemoryManager: Orphaned chunk of 104 bytes found during finalize 9. 2017-07-05 04:31:52,915 INFO [ip-172-31-3-154.ap-northeast-1.compute.internal,16020,1499227309930-index-writer--pool3-t13] regionserver.HRegion: writing data to region TRB,\x0D\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1499228749864.9eca4d9d19a712533aa240853173c816. with WAL disabled. Data may be lost in the event of a crash. 2017-07-05 04:32:04,506 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=11.14 MB, freeSize=10.56 GB, max=10.57 GB, blockCount=4, accesses=26651, hits=26651, hitRatio=100.00%, , cachingAccesses=26651, cachingHits=26651, cachingHitsRatio=100.00%, evictions=179, evicted=0, evictedPerRun=0.0 2017-07-05 04:32:04,556 INFO [BucketCacheStatsExecutor] bucket.BucketCache: failedBlockAdditions=0, totalSize=80.00 GB, freeSize=80.00 GB, usedSize=2.85 MB, cacheSize=1.85 MB, accesses=334, hits=134, IOhitsPerSecond=0, IOTimePerHit=0.15, hitRatio=40.12%, cachingAccesses=239, cachingHits=134, cachingHitsRatio=56.07%, evictions=0, evicted=0, evictedPerRun=NaN 2017-07-05 04:33:33,010 WARN [Finalizer] memory.GlobalMemoryManager: Orphaned chunk of 104 bytes found during finalize 2017-07-05 04:33:40,823 WARN [Finalizer] memory.GlobalMemoryManager: Orphaned chunk of 104 bytes found during finalize 10. 2017-07-05 04:31:53,653 INFO [ip-172-31-7-66.ap-northeast-1.compute.internal,16020,1499227313375-index-writer--pool38-t18] regionserver.HRegion: writing data to region TRP,\x02\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1499228698442.6706144c5be72f993a69ad9ad63005e5. with WAL disabled. Data may be lost in the event of a crash. 2017-07-05 04:32:04,504 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=11.14 MB, freeSize=10.56 GB, max=10.57 GB, blockCount=1, accesses=24493, hits=24493, hitRatio=100.00%, , cachingAccesses=24493, cachingHits=24493, cachingHitsRatio=100.00%, evictions=179, evicted=0, evictedPerRun=0.0 2017-07-05 04:32:04,577 INFO [BucketCacheStatsExecutor] bucket.BucketCache: failedBlockAdditions=0, totalSize=80.00 GB, freeSize=80.00 GB, usedSize=2.48 MB, cacheSize=1.62 MB, accesses=260, hits=81, IOhitsPerSecond=0, IOTimePerHit=0.10, hitRatio=31.15%, cachingAccesses=164, cachingHits=81, cachingHitsRatio=49.39%, evictions=0, evicted=0, evictedPerRun=NaN 2017-07-05 04:36:04,834 WARN [Finalizer] memory.GlobalMemoryManager: Orphaned chunk of 104 bytes found during finalize 11. 2017-07-05 04:31:53,327 INFO [ip-172-31-8-18.ap-northeast-1.compute.internal,16020,1499227319500-index-writer--pool26-t18] regionserver.HRegion: writing data to region TRP,\x0F\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1499228698442.53d87d4772b4e637b4dfd426bb11d624. with WAL disabled. Data may be lost in the event of a crash. 2017-07-05 04:32:04,505 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=11.14 MB, freeSize=10.56 GB, max=10.57 GB, blockCount=1, accesses=23309, hits=23309, hitRatio=100.00%, , cachingAccesses=23309, cachingHits=23309, cachingHitsRatio=100.00%, evictions=179, evicted=0, evictedPerRun=0.0 2017-07-05 04:32:04,574 INFO [BucketCacheStatsExecutor] bucket.BucketCache: failedBlockAdditions=0, totalSize=80.00 GB, freeSize=80.00 GB, usedSize=2.83 MB, cacheSize=1.85 MB, accesses=330, hits=138, IOhitsPerSecond=0, IOTimePerHit=0.12, hitRatio=41.82%, cachingAccesses=232, cachingHits=138, cachingHitsRatio=59.48%, evictions=0, evicted=0, evictedPerRun=NaN 2017-07-05 04:33:58,658 WARN [Finalizer] memory.GlobalMemoryManager: Orphaned chunk of 104 bytes found during finalize The TRP table is mutable which have composite parimary key, and have 5 local indexes. I always upsert data from phoenix only. Phoenix Configuration is below, phoenix.schema.dropMetaData false phoenix.query.rowKeyOrderSaltedTable true hbase.regionserver.wal.codec org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec hbase.region.server.rpc.scheduler.factory.class org.apache.hadoop.hbase.ipc.PhoenixRpcSchedulerFactory phoenix.index.failure.block.write true How can i solve this? Please advise me if you do not mind. Thanks, Takashi
