Martin,
Might you be able to upgrade to a newer version of HDP (and implicitly,
Phoenix)? The default HBase call queue size is 1GB which implies that
things are getting quite backed up for you to see an error like that.
I'd confidently venture a guess that UPSERT SELECT logic is better in
newer versions of the code.
Otherwise, you're in for a treat of trying to understand if there's some
kind of deadlock going on server-side which is preventing the accepted
RPCs from being worked off in a reasonable amount of time.
On 3/26/18 2:18 AM, Hengesbach, Martin wrote:
Hi,
I tried to copy round about 1 million records from one table to another
with this sql:
UPSERT INTO New_Table (ID, data) SELECT ID, data FROM OLD_Table WHERE id
>= '2010000000' AND ID < '2011000000';
With smaller number of records it works, but with this sql I get the
following exception:
/18/03/26 07:54:02 WARN client.AsyncProcess: #1, table=New_Table,
attempt=35/35 failed=28363ops, last exception:
org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.ipc.RpcServer$CallQueueTooBigException):
*Call queue is full* on /x.x.x.x:16020, is
*hbase.ipc.server.max.callqueue.size too small?* on
dn13,16020,1521542835853, tracking started Mon Mar 26 07:44:16 CEST
2018; not retrying 28363 - final failure/
/Error:
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
Failed 28363 actions: RemoteWithExtrasException: 28363 times,
(state=,code=0)/
/org.apache.phoenix.execute.CommitException:
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
Failed 28363 actions: RemoteWithExtrasException: 28363 times,/
/ at
org.apache.phoenix.execute.MutationState.send(MutationState.java:985)/
/ at
org.apache.phoenix.execute.MutationState.send(MutationState.java:1327)/
/ at
org.apache.phoenix.execute.MutationState.commit(MutationState.java:1159)/
/ at
org.apache.phoenix.jdbc.PhoenixConnection$3.call(PhoenixConnection.java:528)/
/ at
org.apache.phoenix.jdbc.PhoenixConnection$3.call(PhoenixConnection.java:525)/
/ at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)/
/ at
org.apache.phoenix.jdbc.PhoenixConnection.commit(PhoenixConnection.java:525)/
/ at
org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:346)/
/ at
org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:331)/
/ at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)/
/ at
org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:329)/
/ at
org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1440)/
/ at sqlline.Commands.execute(Commands.java:822)/
/ at sqlline.Commands.sql(Commands.java:732)/
/ at sqlline.SqlLine.dispatch(SqlLine.java:808)/
/ at sqlline.SqlLine.begin(SqlLine.java:681)/
/ at sqlline.SqlLine.start(SqlLine.java:398)/
/ at sqlline.SqlLine.main(SqlLine.java:292)/
/Caused by:
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
Failed 28363 actions: RemoteWithExtrasException: 28363 times,/
/ at
org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:234)/
/ at
org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1700(AsyncProcess.java:214)/
/ at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.getErrors(AsyncProcess.java:1623)/
/ at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:909)/
/ at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:923)/
/ at
org.apache.phoenix.execute.MutationState.send(MutationState.java:952)/
/ ... 17 more/
I have searched a lot for correct / better values for
*/hbase.ipc.server.max.callqueue.size /*but I don’t find anything. And
in our configuration there is no entry for that.
I tried this on a Hortonworks 2.5.3 with Phoenix 4.7.
Any ideas where I must tune to get this sql running?
Thank’s in advance
Martin
------------------------------------------------------------------------
FIZ Karlsruhe - Leibniz-Institut für Informationsinfrastruktur GmbH.
Sitz der Gesellschaft: Eggenstein-Leopoldshafen, Amtsgericht Mannheim
HRB 101892.
Geschäftsführerin: Sabine Brünger-Weilandt.
Vorsitzender des Aufsichtsrats: MinDirig Dr. Stefan Luther.
