[jira] [Resolved] (PHOENIX-4631) PhoenixInputFormat should close connection after generateSplits()
[ https://issues.apache.org/jira/browse/PHOENIX-4631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hui Zheng resolved PHOENIX-4631. Resolution: Duplicate > PhoenixInputFormat should close connection after generateSplits() > - > > Key: PHOENIX-4631 > URL: https://issues.apache.org/jira/browse/PHOENIX-4631 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.10.0 >Reporter: Hui Zheng >Priority: Major > > In our sparkstreaming usecase which loads a phoenix table as a Dataset, it > will leak zookeeper connection(3 connections per batch) and lead to OOM > exception in its driver process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-4631) PhoenixInputFormat should close connection after generateSplits()
[ https://issues.apache.org/jira/browse/PHOENIX-4631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379681#comment-16379681 ] Hui Zheng commented on PHOENIX-4631: It may be related to [PHOENIX-4319|https://issues.apache.org/jira/browse/PHOENIX-4319] . > PhoenixInputFormat should close connection after generateSplits() > - > > Key: PHOENIX-4631 > URL: https://issues.apache.org/jira/browse/PHOENIX-4631 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.10.0 >Reporter: Hui Zheng >Priority: Major > > In our sparkstreaming usecase which loads a phoenix table as a Dataset, it > will leak zookeeper connection(3 connections per batch) and lead to OOM > exception in its driver process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-4631) PhoenixInputFormat should close connection after generateSplits()
[
https://issues.apache.org/jira/browse/PHOENIX-4631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379674#comment-16379674
]
Hui Zheng commented on PHOENIX-4631:
phoenix-core/src/main/java/org/apache/phoenix/mapreduce/PhoenixInputFormat.java
I think the connection should be cloesd after splits get generated here.
{code:java}
private List generateSplits(final QueryPlan qplan, final
List splits, Configuration config) throws IOException {
Preconditions.checkNotNull(qplan);
Preconditions.checkNotNull(splits);
// Get the RegionSizeCalculator
try(org.apache.hadoop.hbase.client.Connection connection =
HBaseFactoryProvider.getHConnectionFactory().createConnection(config)) {
RegionLocator regionLocator =
connection.getRegionLocator(TableName.valueOf(qplan
.getTableRef().getTable().getPhysicalName().toString()));
RegionSizeCalculator sizeCalculator = new RegionSizeCalculator(regionLocator,
connection
.getAdmin());
{code}
> PhoenixInputFormat should close connection after generateSplits()
> -
>
> Key: PHOENIX-4631
> URL: https://issues.apache.org/jira/browse/PHOENIX-4631
> Project: Phoenix
> Issue Type: Bug
>Affects Versions: 4.10.0
>Reporter: Hui Zheng
>Priority: Major
>
> In our sparkstreaming usecase which loads a phoenix table as a Dataset, it
> will leak zookeeper connection(3 connections per batch) and lead to OOM
> exception in its driver process.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (PHOENIX-4631) PhoenixInputFormat should close connection after generateSplits()
Hui Zheng created PHOENIX-4631: -- Summary: PhoenixInputFormat should close connection after generateSplits() Key: PHOENIX-4631 URL: https://issues.apache.org/jira/browse/PHOENIX-4631 Project: Phoenix Issue Type: Bug Affects Versions: 4.10.0 Reporter: Hui Zheng In our sparkstreaming usecase which loads a phoenix table as a Dataset, it will leak zookeeper connection(3 connections per batch) and lead to OOM exception in its driver process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
