Re: Getting ArrayIndexOutOfBoundsException in UPSERT SELECT
Hi Jaime, I've run into similar issues with Phoenix 4.2.x. They seem to have gone away for me as of 4.3.1. Are you able to upgrade to that version or higher and try those same queries? Josh On Wed, Aug 26, 2015 at 3:30 PM, Jaime Solano jdjsol...@gmail.com wrote: Hi Yiannis, Not quite, but it's similar. For instance, in my case the tables are not salted. I forgot to add, I'm using Phoenix 4.2.0. Thanks for your reply! -Jaime On Wed, Aug 26, 2015 at 3:20 PM, Yiannis Gkoufas johngou...@gmail.com wrote: Hi Jaime, is this https://issues.apache.org/jira/browse/PHOENIX-2169 the error you are getting? Thanks On 26 August 2015 at 19:52, Jaime Solano jdjsol...@gmail.com wrote: Hi guys, I'm getting *Error: java.lang.ArrayIndexOutOfBoundsException (state=08000, code=101)*, while doing a query like the following: UPSERT INTO A (PK, COL2) SELECT A.PK, A.COL1 * B.COL1 FROM A LEFT JOIN B ON A.COL3=B.COL2; Some additional info: - Table A has around 140k records, while table B only has around 100. - In reality, Table A has around 140 columns, but they're not part of the UPSERT in any way. - A.COL2 is DECIMAL. - A.COL1 is FLOAT - B.COL1 is DECIMAL. *- Running only the SELECT statement, successfully prints the results on the screen.* To be honest, I have no idea what to look for or where. Any ideas? Thanks, -Jaime
RE: Error thrown when trying to drop table
OK, and then delete the table within HBase as well I assume? From: James Taylor [mailto:jamestay...@apache.org] Sent: Tuesday, August 25, 2015 6:18 PM To: user Subject: Re: Error thrown when trying to drop table I think you'll need to resort to manually deleting from the SYSTEM.CATALOG table. Something like the following: DELETE FROM SYSTEM.CATALOG WHERE TENANT_ID IS NULL AND TABLE_SCHEM = 'MYSCHEMA' AND TABLE_NAME = 'MYTABLE'; You'll need to bounce your cluster to cause the server-side cache to be cleared too. On Tue, Aug 25, 2015 at 10:29 AM, Michael McAllister mmcallis...@homeaway.commailto:mmcallis...@homeaway.com wrote: James We’re on Phoenix 4.2 on HDP 2.2.6. I’ve tried reproducing the problem and I can’t. My issue is specific to these two tables. When I try recreating the exact same table I can drop it. I’m willing to do some digging on these tables if it will help you, but in the end what I’d like to do is get them dropped. Michael McAllister Staff Data Warehouse Engineer | Decision Systems mmcallis...@homeaway.commailto:mmcallis...@homeaway.com | C: 512.423.7447tel:512.423.7447 | skype: michael.mcallister.hamailto:zimmk...@hotmail.com | webex: https://h.a/mikewebex [cid:image001.png@01D0DFE4.60ED0CF0] This electronic communication (including any attachment) is confidential. If you are not an intended recipient of this communication, please be advised that any disclosure, dissemination, distribution, copying or other use of this communication or any attachment is strictly prohibited. If you have received this communication in error, please notify the sender immediately by reply e-mail and promptly destroy all electronic and printed copies of this communication and any attachment. On Aug 24, 2015, at 10:48 PM, James Taylor jamestay...@apache.orgmailto:jamestay...@apache.org wrote: Hi Michael, What version are you on over there? Can you setup a test that consistently reproduces the issue? Thanks, James On Mon, Aug 24, 2015 at 8:20 PM, Michael McAllister mmcallis...@homeaway.commailto:mmcallis...@homeaway.com wrote: Hi I have an empty table that, when I try and drop, I get an error returned. I issue a command similar to the following:- drop table myschema.mytable; I get the following error:- 22:17:53 [DROP - 0 row(s), 0.276 secs] [Error Code: 101, SQL State: 08000] org.apache.hadoop.hbase.DoNotRetryIOException: MYSCHEMA.MYTABLE: 29 at org.apache.phoenix.util.ServerUtil.createIOException(ServerUtil.java:84) at org.apache.phoenix.coprocessor.MetaDataEndpointImpl.dropTable(MetaDataEndpointImpl.java:965) at org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService.callMethod(MetaDataProtos.java:7768) at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:6896) at org.apache.hadoop.hbase.regionserver.HRegionServer.execServiceOnRegion(HRegionServer.java:3420) at org.apache.hadoop.hbase.regionserver.HRegionServer.execService(HRegionServer.java:3402) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29998) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2078) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.ArrayIndexOutOfBoundsException: 29 at org.apache.phoenix.schema.PTableImpl.init(PTableImpl.java:320) at org.apache.phoenix.schema.PTableImpl.init(PTableImpl.java:250) at org.apache.phoenix.schema.PTableImpl.makePTable(PTableImpl.java:240) at org.apache.phoenix.coprocessor.MetaDataEndpointImpl.getTable(MetaDataEndpointImpl.java:638) at org.apache.phoenix.coprocessor.MetaDataEndpointImpl.buildTable(MetaDataEndpointImpl.java:376) at org.apache.phoenix.coprocessor.MetaDataEndpointImpl.doDropTable(MetaDataEndpointImpl.java:985) at org.apache.phoenix.coprocessor.MetaDataEndpointImpl.dropTable(MetaDataEndpointImpl.java:939) ... 10 more ... 1 statement(s) executed, 0 row(s) affected, exec/fetch time: 0.276/0.000 sec [0 successful, 0 warnings, 1 errors] Michael McAllister Staff Data Warehouse Engineer | Decision Systems mmcallis...@homeaway.commailto:mmcallis...@homeaway.com | C: 512.423.7447tel:512.423.7447 | skype: michael.mcallister.hamailto:zimmk...@hotmail.com | webex: https://h.a/mikewebex image002.png This electronic communication (including any attachment) is confidential. If you are not an intended recipient of this communication, please be advised that any disclosure, dissemination, distribution, copying or other use of this communication or any attachment is strictly prohibited. If you have received this communication in error, please notify the sender immediately by reply e-mail and promptly destroy all electronic and printed copies of this communication and any attachment.
Re: select * from table throws scanner timeout
Glad to know the patch worked. It should be part of our 4.5.2 patch release which we hopefully will be releasing soon. Out of curiosity - did your order by query ever finish? How much time did it take? Also how much time did the original query take with the patch that I provided? On Wed, Aug 26, 2015 at 6:02 AM, Sunil B bsunil...@gmail.com wrote: Hi Samarth, The patch definitely solves the issue. The query select * from table retrieves all the records. Thanks for the patch. Thanks, Sunil On Tue, Aug 25, 2015 at 1:21 PM, Samarth Jain samarth.j...@gmail.com wrote: Thanks for the debugging info Sunil. I have uploaded a possible patch for the issue you are running into - https://issues.apache.org/jira/browse/PHOENIX-2207 Mind giving it a try? On Tue, Aug 25, 2015 at 11:18 AM, Sunil B bsunil...@gmail.com wrote: Hi Samarth, Answers to your questions: 1) How many regions are there? Ans: Total regions: 21. Each region is 5GB uncompressed (1.7GB compressed). Total region servers: 7 2) Do you have phoenix stats enabled? http://phoenix.apache.org/update_statistics.html Ans: The configuration is at default. In my understanding stats are enabled by default. While debugging on the client I did notice that phoenix client divides the query into ~1600 parallel scans using guideposts. Let me know if this doesn't answer your question. 3) Is the table salted? Ans: No 4) Do you have any overrides for scanner caching (hbase.client.scanner.caching) or result size (hbase.client.scanner.max.result.size) in your hbase-site.xml? Ans: No. It is all at default configuration. My Analysis: As I said, it looks like a bug to me. Please let me know if you think otherwise. The chain of Iterators on the client side got from debugging: RoundRobingResultIterator contains 1 ConcatResultIterator. The ConcatResultIterator contains ~1600 LookAheadResultIterator. Each of the LookAheadResultIterator contains one TableResultIterator In ParallelIterators.submitWork() function iterator.peek() is called on each of the 1600 LookAheadResultIterators. This peek() function retrieves data to cache from all the 1600 scanners. After this these LookAheadResultIterators are added to ConcatResultIterator in BaseResultIterators.getIterators() function. As ConcatResultIterator goes serially through the rows, the purpose of peek() function is lost. By the time ConcatResultIterator finishes with first couple of LookAheadResultIterators, the scanners, which were initialized due to peek(), timeout on the Region-Servers. Workaround for now: I am trying the query by adding an order by clause: select * from TABLE order by PRIMARY_KEY This modification seems to be working. It has been running for the past 5 hours now. Will update the thread with success or failure. Code Analysis: ScanPlan.newIterator function uses SerialIterators instead of ParallelIterators if there is an order by in the query. Thanks, Sunil On Mon, Aug 24, 2015 at 11:08 PM, Samarth Jain sama...@apache.org wrote: Sunil, Can you tells us a little bit more about the table - 1) How many regions are there? 2) Do you have phoenix stats enabled? http://phoenix.apache.org/update_statistics.html 3) Is the table salted? 4) Do you have any overrides for scanner caching (hbase.client.scanner.caching) or result size (hbase.client.scanner.max.result.size) in your hbase-site.xml? Thanks, Samarth On Mon, Aug 24, 2015 at 2:03 PM, Sunil B bsunil...@gmail.com wrote: Hi, Phoenix Version: 4.5.0-Hbase-1.0 Client: slqline/JDBC driver I have a large table which has around 100GB of data. I am trying to execute a simple query select * from TABLE, which times out with scanner timeout exception. Please let me know if there is a way to avoid this timeout without changing server side scanner timeout. Exception: WARN client.ScannerCallable: Ignore, probably already closed org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Name: 15791, already closed? The reason for the timeout is that phoenix divides this query into multiple parallel scans and executes scanner.next on each one of them at the start of the query execution (this is because of the use of PeekingResultIterator.peek function being called in submitWork function of the ParallelIterators class). Is there a way I can force Phoenix to do a serial scan instead of parallel scan with PeekingResultIterator? Thanks, Sunil
Re: select * from table throws scanner timeout
Hi Samarth, The patch definitely solves the issue. The query select * from table retrieves all the records. Thanks for the patch. Thanks, Sunil On Tue, Aug 25, 2015 at 1:21 PM, Samarth Jain samarth.j...@gmail.com wrote: Thanks for the debugging info Sunil. I have uploaded a possible patch for the issue you are running into - https://issues.apache.org/jira/browse/PHOENIX-2207 Mind giving it a try? On Tue, Aug 25, 2015 at 11:18 AM, Sunil B bsunil...@gmail.com wrote: Hi Samarth, Answers to your questions: 1) How many regions are there? Ans: Total regions: 21. Each region is 5GB uncompressed (1.7GB compressed). Total region servers: 7 2) Do you have phoenix stats enabled? http://phoenix.apache.org/update_statistics.html Ans: The configuration is at default. In my understanding stats are enabled by default. While debugging on the client I did notice that phoenix client divides the query into ~1600 parallel scans using guideposts. Let me know if this doesn't answer your question. 3) Is the table salted? Ans: No 4) Do you have any overrides for scanner caching (hbase.client.scanner.caching) or result size (hbase.client.scanner.max.result.size) in your hbase-site.xml? Ans: No. It is all at default configuration. My Analysis: As I said, it looks like a bug to me. Please let me know if you think otherwise. The chain of Iterators on the client side got from debugging: RoundRobingResultIterator contains 1 ConcatResultIterator. The ConcatResultIterator contains ~1600 LookAheadResultIterator. Each of the LookAheadResultIterator contains one TableResultIterator In ParallelIterators.submitWork() function iterator.peek() is called on each of the 1600 LookAheadResultIterators. This peek() function retrieves data to cache from all the 1600 scanners. After this these LookAheadResultIterators are added to ConcatResultIterator in BaseResultIterators.getIterators() function. As ConcatResultIterator goes serially through the rows, the purpose of peek() function is lost. By the time ConcatResultIterator finishes with first couple of LookAheadResultIterators, the scanners, which were initialized due to peek(), timeout on the Region-Servers. Workaround for now: I am trying the query by adding an order by clause: select * from TABLE order by PRIMARY_KEY This modification seems to be working. It has been running for the past 5 hours now. Will update the thread with success or failure. Code Analysis: ScanPlan.newIterator function uses SerialIterators instead of ParallelIterators if there is an order by in the query. Thanks, Sunil On Mon, Aug 24, 2015 at 11:08 PM, Samarth Jain sama...@apache.org wrote: Sunil, Can you tells us a little bit more about the table - 1) How many regions are there? 2) Do you have phoenix stats enabled? http://phoenix.apache.org/update_statistics.html 3) Is the table salted? 4) Do you have any overrides for scanner caching (hbase.client.scanner.caching) or result size (hbase.client.scanner.max.result.size) in your hbase-site.xml? Thanks, Samarth On Mon, Aug 24, 2015 at 2:03 PM, Sunil B bsunil...@gmail.com wrote: Hi, Phoenix Version: 4.5.0-Hbase-1.0 Client: slqline/JDBC driver I have a large table which has around 100GB of data. I am trying to execute a simple query select * from TABLE, which times out with scanner timeout exception. Please let me know if there is a way to avoid this timeout without changing server side scanner timeout. Exception: WARN client.ScannerCallable: Ignore, probably already closed org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Name: 15791, already closed? The reason for the timeout is that phoenix divides this query into multiple parallel scans and executes scanner.next on each one of them at the start of the query execution (this is because of the use of PeekingResultIterator.peek function being called in submitWork function of the ParallelIterators class). Is there a way I can force Phoenix to do a serial scan instead of parallel scan with PeekingResultIterator? Thanks, Sunil
