[jira] [Commented] (PHOENIX-4489) HBase Connection leak in Phoenix MR Jobs
[
https://issues.apache.org/jira/browse/PHOENIX-4489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387441#comment-16387441
]
Jepson commented on PHOENIX-4489:
-
[~karanmehta93] Very nice, i have compiled, tested, and been ok.
> HBase Connection leak in Phoenix MR Jobs
>
>
> Key: PHOENIX-4489
> URL: https://issues.apache.org/jira/browse/PHOENIX-4489
> Project: Phoenix
> Issue Type: Bug
>Reporter: Karan Mehta
>Assignee: Karan Mehta
>Priority: Major
> Fix For: 4.14.0, 5.0.0
>
> Attachments: PHOENIX-4489.001.patch, PHOENIX-4489.002.patch,
> PHOENIX-4489.4.x-HBase-0.98.001.patch
>
>
> Phoenix MR jobs uses a custom class {{PhoenixInputFormat}} to determine the
> splits and the parallelism of the work. The class directly opens up a HBase
> connection, which is not closed after the usage. Independently running MR
> jobs should not have any concern, however jobs that run through Phoenix-Spark
> can cause leak issues if this is left unclosed (since those jobs run as a
> part of same JVM).
> Apart from this, the connection should be instantiated with
> {{HBaseFactoryProvider.getHConnectionFactory()}} instead of the default one.
> It can be useful if a separate client is trying to run jobs and wants to
> provide a custom implementation of {{HConnection}}.
> [~jmahonin] Any ideas?
> [~jamestaylor] [~vincentpoon] Any concerns around this?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-4489) HBase Connection leak in Phoenix MR Jobs
[
https://issues.apache.org/jira/browse/PHOENIX-4489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16337080#comment-16337080
]
Karan Mehta commented on PHOENIX-4489:
--
Pushed the patches to all 4.x and 5.x branches. Thanks for the review guys!
> HBase Connection leak in Phoenix MR Jobs
>
>
> Key: PHOENIX-4489
> URL: https://issues.apache.org/jira/browse/PHOENIX-4489
> Project: Phoenix
> Issue Type: Bug
>Reporter: Karan Mehta
>Assignee: Karan Mehta
>Priority: Major
> Attachments: PHOENIX-4489.001.patch, PHOENIX-4489.002.patch,
> PHOENIX-4489.4.x-HBase-0.98.001.patch
>
>
> Phoenix MR jobs uses a custom class {{PhoenixInputFormat}} to determine the
> splits and the parallelism of the work. The class directly opens up a HBase
> connection, which is not closed after the usage. Independently running MR
> jobs should not have any concern, however jobs that run through Phoenix-Spark
> can cause leak issues if this is left unclosed (since those jobs run as a
> part of same JVM).
> Apart from this, the connection should be instantiated with
> {{HBaseFactoryProvider.getHConnectionFactory()}} instead of the default one.
> It can be useful if a separate client is trying to run jobs and wants to
> provide a custom implementation of {{HConnection}}.
> [~jmahonin] Any ideas?
> [~jamestaylor] [~vincentpoon] Any concerns around this?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-4489) HBase Connection leak in Phoenix MR Jobs
[
https://issues.apache.org/jira/browse/PHOENIX-4489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336970#comment-16336970
]
Hudson commented on PHOENIX-4489:
-
FAILURE: Integrated in Jenkins build Phoenix-master #1919 (See
[https://builds.apache.org/job/Phoenix-master/1919/])
PHOENIX-4489 HBase Connection leak in Phoenix MR Jobs (karanmehta93: rev
65f91a11d1bc2cc4d798ced121f47200bf0fc36c)
* (edit)
phoenix-core/src/main/java/org/apache/phoenix/mapreduce/PhoenixInputFormat.java
> HBase Connection leak in Phoenix MR Jobs
>
>
> Key: PHOENIX-4489
> URL: https://issues.apache.org/jira/browse/PHOENIX-4489
> Project: Phoenix
> Issue Type: Bug
>Reporter: Karan Mehta
>Assignee: Karan Mehta
>Priority: Major
> Attachments: PHOENIX-4489.001.patch, PHOENIX-4489.002.patch
>
>
> Phoenix MR jobs uses a custom class {{PhoenixInputFormat}} to determine the
> splits and the parallelism of the work. The class directly opens up a HBase
> connection, which is not closed after the usage. Independently running MR
> jobs should not have any concern, however jobs that run through Phoenix-Spark
> can cause leak issues if this is left unclosed (since those jobs run as a
> part of same JVM).
> Apart from this, the connection should be instantiated with
> {{HBaseFactoryProvider.getHConnectionFactory()}} instead of the default one.
> It can be useful if a separate client is trying to run jobs and wants to
> provide a custom implementation of {{HConnection}}.
> [~jmahonin] Any ideas?
> [~jamestaylor] [~vincentpoon] Any concerns around this?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-4489) HBase Connection leak in Phoenix MR Jobs
[
https://issues.apache.org/jira/browse/PHOENIX-4489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336784#comment-16336784
]
Karan Mehta commented on PHOENIX-4489:
--
Apart from master branch, will other branches do I need to commit this?
> HBase Connection leak in Phoenix MR Jobs
>
>
> Key: PHOENIX-4489
> URL: https://issues.apache.org/jira/browse/PHOENIX-4489
> Project: Phoenix
> Issue Type: Bug
>Reporter: Karan Mehta
>Assignee: Karan Mehta
>Priority: Major
> Attachments: PHOENIX-4489.001.patch, PHOENIX-4489.002.patch
>
>
> Phoenix MR jobs uses a custom class {{PhoenixInputFormat}} to determine the
> splits and the parallelism of the work. The class directly opens up a HBase
> connection, which is not closed after the usage. Independently running MR
> jobs should not have any concern, however jobs that run through Phoenix-Spark
> can cause leak issues if this is left unclosed (since those jobs run as a
> part of same JVM).
> Apart from this, the connection should be instantiated with
> {{HBaseFactoryProvider.getHConnectionFactory()}} instead of the default one.
> It can be useful if a separate client is trying to run jobs and wants to
> provide a custom implementation of {{HConnection}}.
> [~jmahonin] Any ideas?
> [~jamestaylor] [~vincentpoon] Any concerns around this?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-4489) HBase Connection leak in Phoenix MR Jobs
[
https://issues.apache.org/jira/browse/PHOENIX-4489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336783#comment-16336783
]
Karan Mehta commented on PHOENIX-4489:
--
Thanks [~gjacoby] and [~jmahonin]. Will commit this. Also will file a separate
Jira to resolve this issue in Phoenix-Hive.
> HBase Connection leak in Phoenix MR Jobs
>
>
> Key: PHOENIX-4489
> URL: https://issues.apache.org/jira/browse/PHOENIX-4489
> Project: Phoenix
> Issue Type: Bug
>Reporter: Karan Mehta
>Assignee: Karan Mehta
>Priority: Major
> Attachments: PHOENIX-4489.001.patch, PHOENIX-4489.002.patch
>
>
> Phoenix MR jobs uses a custom class {{PhoenixInputFormat}} to determine the
> splits and the parallelism of the work. The class directly opens up a HBase
> connection, which is not closed after the usage. Independently running MR
> jobs should not have any concern, however jobs that run through Phoenix-Spark
> can cause leak issues if this is left unclosed (since those jobs run as a
> part of same JVM).
> Apart from this, the connection should be instantiated with
> {{HBaseFactoryProvider.getHConnectionFactory()}} instead of the default one.
> It can be useful if a separate client is trying to run jobs and wants to
> provide a custom implementation of {{HConnection}}.
> [~jmahonin] Any ideas?
> [~jamestaylor] [~vincentpoon] Any concerns around this?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-4489) HBase Connection leak in Phoenix MR Jobs
[
https://issues.apache.org/jira/browse/PHOENIX-4489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336717#comment-16336717
]
Josh Mahonin commented on PHOENIX-4489:
---
Thanks for that [~karanmehta93], +1 as well
> HBase Connection leak in Phoenix MR Jobs
>
>
> Key: PHOENIX-4489
> URL: https://issues.apache.org/jira/browse/PHOENIX-4489
> Project: Phoenix
> Issue Type: Bug
>Reporter: Karan Mehta
>Assignee: Karan Mehta
>Priority: Major
> Attachments: PHOENIX-4489.001.patch, PHOENIX-4489.002.patch
>
>
> Phoenix MR jobs uses a custom class {{PhoenixInputFormat}} to determine the
> splits and the parallelism of the work. The class directly opens up a HBase
> connection, which is not closed after the usage. Independently running MR
> jobs should not have any concern, however jobs that run through Phoenix-Spark
> can cause leak issues if this is left unclosed (since those jobs run as a
> part of same JVM).
> Apart from this, the connection should be instantiated with
> {{HBaseFactoryProvider.getHConnectionFactory()}} instead of the default one.
> It can be useful if a separate client is trying to run jobs and wants to
> provide a custom implementation of {{HConnection}}.
> [~jmahonin] Any ideas?
> [~jamestaylor] [~vincentpoon] Any concerns around this?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-4489) HBase Connection leak in Phoenix MR Jobs
[
https://issues.apache.org/jira/browse/PHOENIX-4489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336670#comment-16336670
]
Geoffrey Jacoby commented on PHOENIX-4489:
--
+1, [~karanmehta93]
> HBase Connection leak in Phoenix MR Jobs
>
>
> Key: PHOENIX-4489
> URL: https://issues.apache.org/jira/browse/PHOENIX-4489
> Project: Phoenix
> Issue Type: Bug
>Reporter: Karan Mehta
>Assignee: Karan Mehta
>Priority: Major
> Attachments: PHOENIX-4489.001.patch, PHOENIX-4489.002.patch
>
>
> Phoenix MR jobs uses a custom class {{PhoenixInputFormat}} to determine the
> splits and the parallelism of the work. The class directly opens up a HBase
> connection, which is not closed after the usage. Independently running MR
> jobs should not have any concern, however jobs that run through Phoenix-Spark
> can cause leak issues if this is left unclosed (since those jobs run as a
> part of same JVM).
> Apart from this, the connection should be instantiated with
> {{HBaseFactoryProvider.getHConnectionFactory()}} instead of the default one.
> It can be useful if a separate client is trying to run jobs and wants to
> provide a custom implementation of {{HConnection}}.
> [~jmahonin] Any ideas?
> [~jamestaylor] [~vincentpoon] Any concerns around this?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-4489) HBase Connection leak in Phoenix MR Jobs
[
https://issues.apache.org/jira/browse/PHOENIX-4489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336604#comment-16336604
]
Karan Mehta commented on PHOENIX-4489:
--
[~jmahonin] Added the new patch without whitespace changes.
> HBase Connection leak in Phoenix MR Jobs
>
>
> Key: PHOENIX-4489
> URL: https://issues.apache.org/jira/browse/PHOENIX-4489
> Project: Phoenix
> Issue Type: Bug
>Reporter: Karan Mehta
>Assignee: Karan Mehta
>Priority: Major
> Attachments: PHOENIX-4489.001.patch, PHOENIX-4489.002.patch
>
>
> Phoenix MR jobs uses a custom class {{PhoenixInputFormat}} to determine the
> splits and the parallelism of the work. The class directly opens up a HBase
> connection, which is not closed after the usage. Independently running MR
> jobs should not have any concern, however jobs that run through Phoenix-Spark
> can cause leak issues if this is left unclosed (since those jobs run as a
> part of same JVM).
> Apart from this, the connection should be instantiated with
> {{HBaseFactoryProvider.getHConnectionFactory()}} instead of the default one.
> It can be useful if a separate client is trying to run jobs and wants to
> provide a custom implementation of {{HConnection}}.
> [~jmahonin] Any ideas?
> [~jamestaylor] [~vincentpoon] Any concerns around this?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-4489) HBase Connection leak in Phoenix MR Jobs
[
https://issues.apache.org/jira/browse/PHOENIX-4489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336123#comment-16336123
]
Karan Mehta commented on PHOENIX-4489:
--
[~jamestaylor] I will upload a new patch without whitespace changes and that
should fix this issue. Will try to do it EOD.
> HBase Connection leak in Phoenix MR Jobs
>
>
> Key: PHOENIX-4489
> URL: https://issues.apache.org/jira/browse/PHOENIX-4489
> Project: Phoenix
> Issue Type: Bug
>Reporter: Karan Mehta
>Assignee: Karan Mehta
>Priority: Major
> Attachments: PHOENIX-4489.001.patch
>
>
> Phoenix MR jobs uses a custom class {{PhoenixInputFormat}} to determine the
> splits and the parallelism of the work. The class directly opens up a HBase
> connection, which is not closed after the usage. Independently running MR
> jobs should not have any concern, however jobs that run through Phoenix-Spark
> can cause leak issues if this is left unclosed (since those jobs run as a
> part of same JVM).
> Apart from this, the connection should be instantiated with
> {{HBaseFactoryProvider.getHConnectionFactory()}} instead of the default one.
> It can be useful if a separate client is trying to run jobs and wants to
> provide a custom implementation of {{HConnection}}.
> [~jmahonin] Any ideas?
> [~jamestaylor] [~vincentpoon] Any concerns around this?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-4489) HBase Connection leak in Phoenix MR Jobs
[
https://issues.apache.org/jira/browse/PHOENIX-4489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336118#comment-16336118
]
James Taylor commented on PHOENIX-4489:
---
Where are we at with this one, [~karanmehta93]?
> HBase Connection leak in Phoenix MR Jobs
>
>
> Key: PHOENIX-4489
> URL: https://issues.apache.org/jira/browse/PHOENIX-4489
> Project: Phoenix
> Issue Type: Bug
>Reporter: Karan Mehta
>Assignee: Karan Mehta
>Priority: Major
> Attachments: PHOENIX-4489.001.patch
>
>
> Phoenix MR jobs uses a custom class {{PhoenixInputFormat}} to determine the
> splits and the parallelism of the work. The class directly opens up a HBase
> connection, which is not closed after the usage. Independently running MR
> jobs should not have any concern, however jobs that run through Phoenix-Spark
> can cause leak issues if this is left unclosed (since those jobs run as a
> part of same JVM).
> Apart from this, the connection should be instantiated with
> {{HBaseFactoryProvider.getHConnectionFactory()}} instead of the default one.
> It can be useful if a separate client is trying to run jobs and wants to
> provide a custom implementation of {{HConnection}}.
> [~jmahonin] Any ideas?
> [~jamestaylor] [~vincentpoon] Any concerns around this?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-4489) HBase Connection leak in Phoenix MR Jobs
[
https://issues.apache.org/jira/browse/PHOENIX-4489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16316716#comment-16316716
]
Vincent Poon commented on PHOENIX-4489:
---
[~karanmehta93] I think we should fix this - I didn't mean to imply otherwise.
I was just saying in our particular use case, I wasn't sure if this was the
cause of the issue we're seeing since we don't run that many MR jobs.
> HBase Connection leak in Phoenix MR Jobs
>
>
> Key: PHOENIX-4489
> URL: https://issues.apache.org/jira/browse/PHOENIX-4489
> Project: Phoenix
> Issue Type: Bug
>Reporter: Karan Mehta
>Assignee: Karan Mehta
> Attachments: PHOENIX-4489.001.patch
>
>
> Phoenix MR jobs uses a custom class {{PhoenixInputFormat}} to determine the
> splits and the parallelism of the work. The class directly opens up a HBase
> connection, which is not closed after the usage. Independently running MR
> jobs should not have any concern, however jobs that run through Phoenix-Spark
> can cause leak issues if this is left unclosed (since those jobs run as a
> part of same JVM).
> Apart from this, the connection should be instantiated with
> {{HBaseFactoryProvider.getHConnectionFactory()}} instead of the default one.
> It can be useful if a separate client is trying to run jobs and wants to
> provide a custom implementation of {{HConnection}}.
> [~jmahonin] Any ideas?
> [~jamestaylor] [~vincentpoon] Any concerns around this?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (PHOENIX-4489) HBase Connection leak in Phoenix MR Jobs
[
https://issues.apache.org/jira/browse/PHOENIX-4489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16315655#comment-16315655
]
Karan Mehta commented on PHOENIX-4489:
--
[~vincentpoon]
Technically, as we discussed it shouldn't be a problem since we go out of scope
real quick after the generateSplits() method is executed and the connection
object should be garbage collected. However, if you checkout PHOENIX-4503, the
client is trying to read multiple spark dataframes inside a loop (almost 50
times). Such a code will get executed fast and will result in lots of
HConnections and ZKConnections getting created in a short span of time and I
suspect that even though GC gets triggered to clear them, it might actually
take some time before this to happen (until JVM feels the need). This can cause
issues with the application. I see many issues filed in this regard.
Also, since the connections are not instantiated via factory, it is difficult
to catch their quantity and limit the resources by having a custom
implementation. What do you think?
FYI, [~aertoria]
> HBase Connection leak in Phoenix MR Jobs
>
>
> Key: PHOENIX-4489
> URL: https://issues.apache.org/jira/browse/PHOENIX-4489
> Project: Phoenix
> Issue Type: Bug
>Reporter: Karan Mehta
>Assignee: Karan Mehta
> Attachments: PHOENIX-4489.001.patch
>
>
> Phoenix MR jobs uses a custom class {{PhoenixInputFormat}} to determine the
> splits and the parallelism of the work. The class directly opens up a HBase
> connection, which is not closed after the usage. Independently running MR
> jobs should not have any concern, however jobs that run through Phoenix-Spark
> can cause leak issues if this is left unclosed (since those jobs run as a
> part of same JVM).
> Apart from this, the connection should be instantiated with
> {{HBaseFactoryProvider.getHConnectionFactory()}} instead of the default one.
> It can be useful if a separate client is trying to run jobs and wants to
> provide a custom implementation of {{HConnection}}.
> [~jmahonin] Any ideas?
> [~jamestaylor] [~vincentpoon] Any concerns around this?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (PHOENIX-4489) HBase Connection leak in Phoenix MR Jobs
[
https://issues.apache.org/jira/browse/PHOENIX-4489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16315652#comment-16315652
]
Karan Mehta commented on PHOENIX-4489:
--
Yes [~jmahonin] The issue exists in {{PhoenixInputFormat}} class in
phoenix-hive integration too. I will fix both of them as a part of second patch.
> HBase Connection leak in Phoenix MR Jobs
>
>
> Key: PHOENIX-4489
> URL: https://issues.apache.org/jira/browse/PHOENIX-4489
> Project: Phoenix
> Issue Type: Bug
>Reporter: Karan Mehta
>Assignee: Karan Mehta
> Attachments: PHOENIX-4489.001.patch
>
>
> Phoenix MR jobs uses a custom class {{PhoenixInputFormat}} to determine the
> splits and the parallelism of the work. The class directly opens up a HBase
> connection, which is not closed after the usage. Independently running MR
> jobs should not have any concern, however jobs that run through Phoenix-Spark
> can cause leak issues if this is left unclosed (since those jobs run as a
> part of same JVM).
> Apart from this, the connection should be instantiated with
> {{HBaseFactoryProvider.getHConnectionFactory()}} instead of the default one.
> It can be useful if a separate client is trying to run jobs and wants to
> provide a custom implementation of {{HConnection}}.
> [~jmahonin] Any ideas?
> [~jamestaylor] [~vincentpoon] Any concerns around this?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (PHOENIX-4489) HBase Connection leak in Phoenix MR Jobs
[
https://issues.apache.org/jira/browse/PHOENIX-4489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314149#comment-16314149
]
Ethan Wang commented on PHOENIX-4489:
-
[~karanmehta93]
patch +1
white space comment from [~jmahonin] +1 too. please consider remove them
> HBase Connection leak in Phoenix MR Jobs
>
>
> Key: PHOENIX-4489
> URL: https://issues.apache.org/jira/browse/PHOENIX-4489
> Project: Phoenix
> Issue Type: Bug
>Reporter: Karan Mehta
>Assignee: Karan Mehta
> Attachments: PHOENIX-4489.001.patch
>
>
> Phoenix MR jobs uses a custom class {{PhoenixInputFormat}} to determine the
> splits and the parallelism of the work. The class directly opens up a HBase
> connection, which is not closed after the usage. Independently running MR
> jobs should not have any concern, however jobs that run through Phoenix-Spark
> can cause leak issues if this is left unclosed (since those jobs run as a
> part of same JVM).
> Apart from this, the connection should be instantiated with
> {{HBaseFactoryProvider.getHConnectionFactory()}} instead of the default one.
> It can be useful if a separate client is trying to run jobs and wants to
> provide a custom implementation of {{HConnection}}.
> [~jmahonin] Any ideas?
> [~jamestaylor] [~vincentpoon] Any concerns around this?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (PHOENIX-4489) HBase Connection leak in Phoenix MR Jobs
[
https://issues.apache.org/jira/browse/PHOENIX-4489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313883#comment-16313883
]
Josh Mahonin commented on PHOENIX-4489:
---
[~karanmehta93] Good find. I suspect I introduced this in PHOENIX-3600 when
adding the location awareness. Incidentally, all I did was port that code from
the phoenix-hive integration, so it might be worth validating on that side if
the bug exists there as well.
Your patch seems to be mostly whitespace fixes, which makes it a bit hard to
review. Is it possible to re-submit with whitespace kept as is (and if needed,
fix tab/spacing on a subsequent patch)? Thanks!
> HBase Connection leak in Phoenix MR Jobs
>
>
> Key: PHOENIX-4489
> URL: https://issues.apache.org/jira/browse/PHOENIX-4489
> Project: Phoenix
> Issue Type: Bug
>Reporter: Karan Mehta
>Assignee: Karan Mehta
> Attachments: PHOENIX-4489.001.patch
>
>
> Phoenix MR jobs uses a custom class {{PhoenixInputFormat}} to determine the
> splits and the parallelism of the work. The class directly opens up a HBase
> connection, which is not closed after the usage. Independently running MR
> jobs should not have any concern, however jobs that run through Phoenix-Spark
> can cause leak issues if this is left unclosed (since those jobs run as a
> part of same JVM).
> Apart from this, the connection should be instantiated with
> {{HBaseFactoryProvider.getHConnectionFactory()}} instead of the default one.
> It can be useful if a separate client is trying to run jobs and wants to
> provide a custom implementation of {{HConnection}}.
> [~jmahonin] Any ideas?
> [~jamestaylor] [~vincentpoon] Any concerns around this?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (PHOENIX-4489) HBase Connection leak in Phoenix MR Jobs
[
https://issues.apache.org/jira/browse/PHOENIX-4489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313790#comment-16313790
]
James Taylor commented on PHOENIX-4489:
---
Let's please spread out the load for code reviews. Maybe [~aertoria] or
[~vincentpoon] could take a look?
> HBase Connection leak in Phoenix MR Jobs
>
>
> Key: PHOENIX-4489
> URL: https://issues.apache.org/jira/browse/PHOENIX-4489
> Project: Phoenix
> Issue Type: Bug
>Reporter: Karan Mehta
>Assignee: Karan Mehta
> Attachments: PHOENIX-4489.001.patch
>
>
> Phoenix MR jobs uses a custom class {{PhoenixInputFormat}} to determine the
> splits and the parallelism of the work. The class directly opens up a HBase
> connection, which is not closed after the usage. Independently running MR
> jobs should not have any concern, however jobs that run through Phoenix-Spark
> can cause leak issues if this is left unclosed (since those jobs run as a
> part of same JVM).
> Apart from this, the connection should be instantiated with
> {{HBaseFactoryProvider.getHConnectionFactory()}} instead of the default one.
> It can be useful if a separate client is trying to run jobs and wants to
> provide a custom implementation of {{HConnection}}.
> [~jmahonin] Any ideas?
> [~jamestaylor] [~vincentpoon] Any concerns around this?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (PHOENIX-4489) HBase Connection leak in Phoenix MR Jobs
[
https://issues.apache.org/jira/browse/PHOENIX-4489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313752#comment-16313752
]
Karan Mehta commented on PHOENIX-4489:
--
FYI. [~tdsilva]
> HBase Connection leak in Phoenix MR Jobs
>
>
> Key: PHOENIX-4489
> URL: https://issues.apache.org/jira/browse/PHOENIX-4489
> Project: Phoenix
> Issue Type: Bug
>Reporter: Karan Mehta
>Assignee: Karan Mehta
> Attachments: PHOENIX-4489.001.patch
>
>
> Phoenix MR jobs uses a custom class {{PhoenixInputFormat}} to determine the
> splits and the parallelism of the work. The class directly opens up a HBase
> connection, which is not closed after the usage. Independently running MR
> jobs should not have any concern, however jobs that run through Phoenix-Spark
> can cause leak issues if this is left unclosed (since those jobs run as a
> part of same JVM).
> Apart from this, the connection should be instantiated with
> {{HBaseFactoryProvider.getHConnectionFactory()}} instead of the default one.
> It can be useful if a separate client is trying to run jobs and wants to
> provide a custom implementation of {{HConnection}}.
> [~jmahonin] Any ideas?
> [~jamestaylor] [~vincentpoon] Any concerns around this?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (PHOENIX-4489) HBase Connection leak in Phoenix MR Jobs
[
https://issues.apache.org/jira/browse/PHOENIX-4489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16302889#comment-16302889
]
Hadoop QA commented on PHOENIX-4489:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12903551/PHOENIX-4489.001.patch
against master branch at commit 34693843abe4490b54fbd30512bf7d98d0f59c0d.
ATTACHMENT ID: 12903551
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 release audit{color}. The applied patch generated 1 release
audit warnings (more than the master's current 0 warnings).
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+.getAttributesMap() + " [scanCache,
cacheBlock, scanBatch] : [" +
+aScan.getCaching() + ", " +
aScan.getCacheBlocks() + ", " + aScan
+psplits.add(new
PhoenixInputSplit(Collections.singletonList(aScan), regionSize,
regionLocation));
+.get(0).getStartRow()) + " ~ " +
Bytes.toStringBinary(scans.get(scans
+.get(0).getAttributesMap() + " [scanCache,
cacheBlock, scanBatch] : " +
+"[" + scans.get(0).getCaching() + ", " +
scans.get(0).getCacheBlocks()
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1689//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1689//artifact/patchprocess/patchReleaseAuditWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-PHOENIX-Build/1689//console
This message is automatically generated.
> HBase Connection leak in Phoenix MR Jobs
>
>
> Key: PHOENIX-4489
> URL: https://issues.apache.org/jira/browse/PHOENIX-4489
> Project: Phoenix
> Issue Type: Bug
>Reporter: Karan Mehta
>Assignee: Karan Mehta
> Attachments: PHOENIX-4489.001.patch
>
>
> Phoenix MR jobs uses a custom class {{PhoenixInputFormat}} to determine the
> splits and the parallelism of the work. The class directly opens up a HBase
> connection, which is not closed after the usage. Independently running MR
> jobs should not have any concern, however jobs that run through Phoenix-Spark
> can cause leak issues if this is left unclosed (since those jobs run as a
> part of same JVM).
> Apart from this, the connection should be instantiated with
> {{HBaseFactoryProvider.getHConnectionFactory()}} instead of the default one.
> It can be useful if a separate client is trying to run jobs and wants to
> provide a custom implementation of {{HConnection}}.
> [~jmahonin] Any ideas?
> [~jamestaylor] [~vincentpoon] Any concerns around this?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (PHOENIX-4489) HBase Connection leak in Phoenix MR Jobs
[
https://issues.apache.org/jira/browse/PHOENIX-4489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16302171#comment-16302171
]
James Taylor commented on PHOENIX-4489:
---
Yes, definitely a concern. Let's make sure there are no connection leaks. This
may be the cause of the reported issue over on PHOENIX-4247. Perhaps this is a
duplicate?
FYI, [~kumarappan].
> HBase Connection leak in Phoenix MR Jobs
>
>
> Key: PHOENIX-4489
> URL: https://issues.apache.org/jira/browse/PHOENIX-4489
> Project: Phoenix
> Issue Type: Bug
>Reporter: Karan Mehta
>
> Phoenix MR jobs uses a custom class {{PhoenixInputFormat}} to determine the
> splits and the parallelism of the work. The class directly opens up a HBase
> connection, which is not closed after the usage. Independently running MR
> jobs should not have any concern, however jobs that run through Phoenix-Spark
> can cause leak issues if this is left unclosed (since those jobs run as a
> part of same JVM).
> Apart from this, the connection should be instantiated with
> {{HBaseFactoryProvider.getHConnectionFactory()}} instead of the default one.
> It can be useful if a separate client is trying to run jobs and wants to
> provide a custom implementation of {{HConnection}}.
> [~jmahonin] Any ideas?
> [~jamestaylor] [~vincentpoon] Any concerns around this?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (PHOENIX-4489) HBase Connection leak in Phoenix MR Jobs
[
https://issues.apache.org/jira/browse/PHOENIX-4489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16302159#comment-16302159
]
Karan Mehta commented on PHOENIX-4489:
--
The fix seems straight forward but want to make sure its correct before moving
on.
> HBase Connection leak in Phoenix MR Jobs
>
>
> Key: PHOENIX-4489
> URL: https://issues.apache.org/jira/browse/PHOENIX-4489
> Project: Phoenix
> Issue Type: Bug
>Reporter: Karan Mehta
>
> Phoenix MR jobs uses a custom class {{PhoenixInputFormat}} to determine the
> splits and the parallelism of the work. The class directly opens up a HBase
> connection, which is not closed after the usage. Independently running MR
> jobs should not have any concern, however jobs that run through Phoenix-Spark
> can cause leak issues if this is left unclosed (since those jobs run as a
> part of same JVM).
> Apart from this, the connection should be instantiated with
> {[HBaseFactoryProvider.getHConnectionFactory()}} instead of the default one.
> It can be useful if a separate client is trying to run jobs and wants to
> provide a custom implementation of {{HConnection}}.
> [~jmahonin] Any ideas?
> [~jamestaylor] [~vincentpoon] Any concerns around this?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
