Hi, who may help
We are running a Hadoop application that needs to use phoenix JDBC connection
from the workers.
The connection works, but when too many connection established at the same
time, it throws RPC timeouts
Error: java.io.IOException: org.apache.phoenix.exception.PhoenixIOException:
Failed after attempts=36, exceptions: Wed Feb 20 20:02:43 UTC 2019, null,
java.net.SocketTimeoutException: callTimeout=60000, callDuration=60506. ...
So we have figured we should probably set a higher hbase.rpc.timeout value,
but then it comes to the issue:
A little bit background on how we run the application
Here is how we get PhoenixConnection from java program
DriverManager.getConnection("jdbc:phoenix:host", props)
And we trigger the program by using
hadoop jar $test_jar
We have tried multiple approaches to load hbase/phoenix configuration, but none
of them get respected by PhoenixConnection, here are the methods we tried
* Pass hbase_conf_dir through HADOOP_CLASSPATH, so run the hadoop application
like HADOOP_CLASSPATH=/etc/hbase/conf/ hadoop jar $test_jar . However,
PhoenixConnection doesn’t respect the parameters
* Tried passing -Dhbase.rpc.timeout=1800, which is picked up by hbase conf
object, but not PhoniexConnection
* Explicitly set those parameters and pass them to the PhoenixConnection
props.setProperty("hbase.rpc.timeout", "1800");
props.setProperty(“phoenix.query.timeoutMs", "1800");
Also didn’t get respected by PhoenixConnection
* also tried what is suggested by phoenix here
https://phoenix.apache.org/#connStr , use :longRunning together with those
properties, still didn’t seem to work
Besides all those approaches we tried, I have explicitly output those
parameters we care from the connection,
connection.getQueryServices().getProps()
The default values I got are 60000 for hbase.rpc.timeout, and 600k for
phoenix.query.timeoutMs , so I have tried to run a query lthat would run longer
than 10 mins, Ideally it should timeout, however, it runs over 20 mins and
didn’t timeout. So I’m wondering how PhoenixConnection respect those properties?
So with some of your help, we’d like to know if there’s any thing wrong with
our approaches. And we’d like to get rid of those SocketTimeExceptions.
We are using phoenix-core version is 4.7.0-clabs-phoenix1.3.0 , and our
phoenix-client version is phoenix-4.7.0-clabs-phoenix1.3.0.23 (we have tried
phoenix-4.14.0-HBase-1.3 as well, which didn’t work either).
Thanks for your time
