Re: Phoenix Exception - Connection is null or closed.

[Anil]

Hi Yun,

You mean, we need to increase the connections in connection pool ?. We dint
have any changes to our code before upgrade. Same code is working on 4.7
version.

Note : We are using APACHE_PHOENIX-4.11.0-cdh5.11.2.p0.1.

Thanks,
Anil



On 24 March 2018 at 20:57, zhang yun  wrote:

> Hi, Anil.
> It seems that this exception is not bug,  you should check your code of
> logical layer. Sometimes this exception will be trigger when multiple
> thread to access connections pool( some connections are closed at that
> time).
>
>
> 
>Yun Zhang
>Best regards!
>
>
> 2018-03-23 23:45 GMT+08:00 Anil :
>
>> Hi Josh,
>>
>> i didn't find any other exception. i will check again. Exceptions
>> happened only on few nodes. i have seen failed to get regions exception on
>> few nodes. I will check agian and comeback to more details. Thanks.
>>
>> Regards,
>> Anil
>>
>> On 22 March 2018 at 21:56, Josh Elser  wrote:
>>
>>> Hey Anil,
>>>
>>> You sure there isn't another exception earlier on in the output of your
>>> application? The exception you have here looks more like the JVM was
>>> already shutting down and Phoenix had closed the connection (the exceptions
>>> were about queued tasks being cleared out after the decision to shutdown
>>> was already made).
>>>
>>> On 3/22/18 8:14 AM, Anil wrote:
>>>
 HI Team,

 We have upgraded the phoenix from 4.7.0 to 4.11.0 and started noticing
 the attached exception.

 Can you help me identifying the root cause of the exception ? Thanks.

 Regards,
 Anil

>>>
>>
>

Re: Phoenix Exception - Connection is null or closed.

[zhang yun]

Hi, Anil.
It seems that this exception is not bug,  you should check your code of
logical layer. Sometimes this exception will be trigger when multiple
thread to access connections pool( some connections are closed at that
time).



   Yun Zhang
   Best regards!


2018-03-23 23:45 GMT+08:00 Anil :

> Hi Josh,
>
> i didn't find any other exception. i will check again. Exceptions happened
> only on few nodes. i have seen failed to get regions exception on few
> nodes. I will check agian and comeback to more details. Thanks.
>
> Regards,
> Anil
>
> On 22 March 2018 at 21:56, Josh Elser  wrote:
>
>> Hey Anil,
>>
>> You sure there isn't another exception earlier on in the output of your
>> application? The exception you have here looks more like the JVM was
>> already shutting down and Phoenix had closed the connection (the exceptions
>> were about queued tasks being cleared out after the decision to shutdown
>> was already made).
>>
>> On 3/22/18 8:14 AM, Anil wrote:
>>
>>> HI Team,
>>>
>>> We have upgraded the phoenix from 4.7.0 to 4.11.0 and started noticing
>>> the attached exception.
>>>
>>> Can you help me identifying the root cause of the exception ? Thanks.
>>>
>>> Regards,
>>> Anil
>>>
>>
>

Re: Phoenix Exception - Connection is null or closed.

[Anil]

Hi Josh,

i didn't find any other exception. i will check again. Exceptions happened
only on few nodes. i have seen failed to get regions exception on few
nodes. I will check agian and comeback to more details. Thanks.

Regards,
Anil

On 22 March 2018 at 21:56, Josh Elser  wrote:

> Hey Anil,
>
> You sure there isn't another exception earlier on in the output of your
> application? The exception you have here looks more like the JVM was
> already shutting down and Phoenix had closed the connection (the exceptions
> were about queued tasks being cleared out after the decision to shutdown
> was already made).
>
> On 3/22/18 8:14 AM, Anil wrote:
>
>> HI Team,
>>
>> We have upgraded the phoenix from 4.7.0 to 4.11.0 and started noticing
>> the attached exception.
>>
>> Can you help me identifying the root cause of the exception ? Thanks.
>>
>> Regards,
>> Anil
>>
>

Re: Phoenix Exception - Connection is null or closed.

[Josh Elser]

Hey Anil,

You sure there isn't another exception earlier on in the output of your 
application? The exception you have here looks more like the JVM was 
already shutting down and Phoenix had closed the connection (the 
exceptions were about queued tasks being cleared out after the decision 
to shutdown was already made).


On 3/22/18 8:14 AM, Anil wrote:

HI Team,

We have upgraded the phoenix from 4.7.0 to 4.11.0 and started noticing 
the attached exception.


Can you help me identifying the root cause of the exception ? Thanks.

Regards,
Anil

Phoenix Exception - Connection is null or closed.

[Anil]

HI Team,

We have upgraded the phoenix from 4.7.0 to 4.11.0 and started noticing the
attached exception.

Can you help me identifying the root cause of the exception ? Thanks.

Regards,
Anil
2018-03-21 08:13:19,684 ERROR 
com.tst.hadoop.flume.writer.inventory.AccountPersistenceImpl: Error querying 
account UUIDs for siteId 0101293035
org.springframework.dao.DataAccessResourceFailureException: 
PreparedStatementCallback; SQL [select uuids from account where siteid = ?]; 
java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException: 
Task 
org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@5ddc0ec5
 rejected from java.util.concurrent.ThreadPoolExecutor@5204afb0[Shutting down, 
pool size = 59, active threads = 0, queued tasks = 0, completed tasks = 
2667625]; nested exception is org.apache.phoenix.exception.PhoenixIOException: 
java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException: 
Task 
org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@5ddc0ec5
 rejected from java.util.concurrent.ThreadPoolExecutor@5204afb0[Shutting down, 
pool size = 59, active threads = 0, queued tasks = 0, completed tasks = 2667625]
at 
org.springframework.jdbc.support.SQLStateSQLExceptionTranslator.doTranslate(SQLStateSQLExceptionTranslator.java:105)
at 
org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:73)
at 
org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81)
at 
org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81)
at 
org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:660)
at 
org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:695)
at 
org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:722)
at 
org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:772)
at 
org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate.query(NamedParameterJdbcTemplate.java:192)
at 
org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate.queryForList(NamedParameterJdbcTemplate.java:290)
at 
com.tst.hadoop.flume.writer.inventory.AccountPersistenceImpl.getUUIDs(AccountPersistenceImpl.java:187)
at 
net.juniper.spark.stream.sap.data.mapper.ServiceContractMapper.populatePartnerAccountId(ServiceContractMapper.java:134)
at 
net.juniper.spark.stream.sap.data.mapper.ServiceContractMapper.map(ServiceContractMapper.java:70)
at 
net.juniper.spark.stream.sap.data.mapper.ServiceContractMapper.map(ServiceContractMapper.java:39)
at 
net.juniper.spark.stream.sap.processor.SAPDataProcessor.mapObject(SAPDataProcessor.java:34)
at 
net.juniper.spark.stream.sap.processor.SAPDataProcessor.processData(SAPDataProcessor.java:43)
at 
com.tst.hadoop.flume.sink.SAPContractCustomSink.process(SAPContractCustomSink.java:113)
at 
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.phoenix.exception.PhoenixIOException: 
java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException: 
Task 
org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@5ddc0ec5
 rejected from java.util.concurrent.ThreadPoolExecutor@5204afb0[Shutting down, 
pool size = 59, active threads = 0, queued tasks = 0, completed tasks = 2667625]
at 
org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:116)
at 
org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:875)
at 
org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:819)
at 
org.apache.phoenix.iterate.RoundRobinResultIterator.getIterators(RoundRobinResultIterator.java:176)
at 
org.apache.phoenix.iterate.RoundRobinResultIterator.next(RoundRobinResultIterator.java:91)
at 
org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:778)
at 
org.apache.commons.dbcp.DelegatingResultSet.next(DelegatingResultSet.java:207)
at 
org.apache.commons.dbcp.DelegatingResultSet.next(DelegatingResultSet.java:207)
at 
org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:92)
at 
org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:60)
at 
org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:708)
at 
org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:644)
... 15 more
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: