Hi I have 2 issues with phoenix client:
1. Heap memory is not cleanup after each query is finished. So, it keeps increasing every time when we submit new query. 2. I am try to do a normal join operation on two tables but getting exception. Below is the details: These are some sample queries I tried: 1. select p1.host, count(1) from PERFORMANCE_5000000 p1, PERFORMANCE_25000000 p2 where p1.host = p2.host group by p1.host; 2. select p1.host from PERFORMANCE_5000000 p1, PERFORMANCE_25000000 p2 where p1.host = p2.host group by p1.host; 3. select count(1) from PERFORMANCE_5000000 p1, PERFORMANCE_25000000 p2 where p1.host = p2.host group by p1.host; Here is explain plan: explain select count(1) from PERFORMANCE_5000000 p1, PERFORMANCE_25000000 p2 where p1.host = p2.host group by p1.host; +------------------------------------------+ | PLAN | +------------------------------------------+ | CLIENT 9-CHUNK PARALLEL 1-WAY FULL SCAN OVER PERFORMANCE_5000000 | | SERVER FILTER BY FIRST KEY ONLY | | SERVER AGGREGATE INTO ORDERED DISTINCT ROWS BY [HOST] | | CLIENT MERGE SORT | | PARALLEL INNER-JOIN TABLE 0 (SKIP MERGE) | | CLIENT 18-CHUNK PARALLEL 1-WAY FULL SCAN OVER PERFORMANCE_25000000 | | SERVER FILTER BY FIRST KEY ONLY | | DYNAMIC SERVER FILTER BY HOST IN (P2.HOST) | +------------------------------------------+ 8 rows selected (0.127 seconds) Phoenix client heap size is 16GB. ( noticed that above queries are dumping data in local heap, I see millions of instances for org.apache.phoenix.expression.literalexpression) and my exceptions are: java.sql.SQLException: Encountered exception in sub plan [0] execution. at org.apache.phoenix.execute.HashJoinPlan.iterator(HashJoinPlan.java:156) at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:235) at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:226) at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53) at org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:225) at org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1066) at sqlline.Commands.execute(Commands.java:822) at sqlline.Commands.sql(Commands.java:732) at sqlline.SqlLine.dispatch(SqlLine.java:808) at sqlline.SqlLine.begin(SqlLine.java:681) at sqlline.SqlLine.start(SqlLine.java:398) at sqlline.SqlLine.main(SqlLine.java:292) Caused by: java.sql.SQLException: java.util.concurrent.ExecutionException: java.lang.Exception: java.lang.OutOfMemoryError: Java heap space at org.apache.phoenix.cache.ServerCacheClient.addServerCache(ServerCacheClient.java:247) at org.apache.phoenix.join.HashCacheClient.addHashCache(HashCacheClient.java:83) at org.apache.phoenix.execute.HashJoinPlan$HashSubPlan.execute(HashJoinPlan.java:338) at org.apache.phoenix.execute.HashJoinPlan$1.call(HashJoinPlan.java:135) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.concurrent.ExecutionException: java.lang.Exception: java.lang.OutOfMemoryError: Java heap space at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:206) at org.apache.phoenix.cache.ServerCacheClient.addServerCache(ServerCacheClient.java:239) ... 7 more Caused by: java.lang.Exception: java.lang.OutOfMemoryError: Java heap space at org.apache.phoenix.cache.ServerCacheClient$1.call(ServerCacheClient.java:212) at org.apache.phoenix.cache.ServerCacheClient$1.call(ServerCacheClient.java:182) ... 4 more Caused by: java.lang.OutOfMemoryError: Java heap space May 20, 2015 4:58:01 PM ServerCommunicatorAdmin reqIncoming WARNING: The server has decided to close this client connection. 15/05/20 16:56:43 WARN client.HTable: Error calling coprocessor service org.apache.phoenix.coprocessor.generated.ServerCachingProtos$ServerCachingService for row CSGoogle\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1620) at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1577) at org.apache.phoenix.cache.ServerCacheClient$1.call(ServerCacheClient.java:188) at org.apache.phoenix.cache.ServerCacheClient$1.call(ServerCacheClient.java:182) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.OutOfMemoryError: Java heap space at com.google.protobuf.ByteString$CodedBuilder.<init>(ByteString.java:907) at com.google.protobuf.ByteString$CodedBuilder.<init>(ByteString.java:902) at com.google.protobuf.ByteString.newCodedBuilder(ByteString.java:898) at com.google.protobuf.AbstractMessageLite.toByteString(AbstractMessageLite.java:49) at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:81) at org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel.callMethod(CoprocessorRpcChannel.java:57) at org.apache.phoenix.coprocessor.generated.ServerCachingProtos$ServerCachingService$Stub.addServerCache(ServerCachingProtos.java:3270) at org.apache.phoenix.cache.ServerCacheClient$1$1.call(ServerCacheClient.java:204) at org.apache.phoenix.cache.ServerCacheClient$1$1.call(ServerCacheClient.java:189) at org.apache.hadoop.hbase.client.HTable$17.call(HTable.java:1608) ... 4 more 0: jdbc:phoenix:prbhadoop004iad.io.askjeeves.> 15/05/20 16:56:43 WARN client.HTable: Error calling coprocessor service org.apache.phoenix.coprocessor.generated.ServerCachingProtos$ServerCachingService for row EUGoogle\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1620) at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1577) at org.apache.phoenix.cache.ServerCacheClient$1.call(ServerCacheClient.java:188) at org.apache.phoenix.cache.ServerCacheClient$1.call(ServerCacheClient.java:182) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.OutOfMemoryError: Java heap space 15/05/20 16:59:37 WARN client.HConnectionManager$HConnectionImplementation: This client just lost it's session with ZooKeeper, closing it. It will be recreated next time someone needs it org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:403) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:321) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) Error: (state=,code=0) java.sql.SQLFeatureNotSupportedException at org.apache.phoenix.jdbc.PhoenixStatement.cancel(PhoenixStatement.java:958) at sqlline.DispatchCallback.forceKillSqlQuery(DispatchCallback.java:83) at sqlline.SqlLine.begin(SqlLine.java:695) at sqlline.SqlLine.start(SqlLine.java:398) at sqlline.SqlLine.main(SqlLine.java:292) Can someone please help? Thanks!
