[jira] [Commented] (HBASE-19848) Zookeeper thread leaks in hbase-spark bulkLoad method
[ https://issues.apache.org/jira/browse/HBASE-19848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351007#comment-16351007 ] Key Hutu commented on HBASE-19848: -- thanks for your help, Ted Yu, huaxiang sun, thanks > Zookeeper thread leaks in hbase-spark bulkLoad method > - > > Key: HBASE-19848 > URL: https://issues.apache.org/jira/browse/HBASE-19848 > Project: HBase > Issue Type: Bug > Components: spark, Zookeeper >Affects Versions: 1.2.0 > Environment: hbase-spark-1.2.0-cdh5.12.1 version > spark 1.6 >Reporter: Key Hutu >Assignee: Key Hutu >Priority: Major > Labels: performance > Attachments: HBASE-19848-V2.patch, HBASE-19848-V3.patch, > HBaseContext.patch, HBaseContext.scala > > Original Estimate: 72h > Remaining Estimate: 72h > > In hbase-spark project, HBaseContext provides bulkload methond for loading > spark rdd data to hbase easily.But when i using it frequently， the program > will throw "cannot create native thread" exception. > using pstack command in spark driver process , the thread num is increasing > using jstack, named "main-SendThread" and "main-EventThread" thread so many > It seems like that , connection created before bulkload ,but close method > uninvoked at last -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19848) Zookeeper thread leaks in hbase-spark bulkLoad method
[ https://issues.apache.org/jira/browse/HBASE-19848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350936#comment-16350936 ] Hudson commented on HBASE-19848: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4515 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4515/]) HBASE-19848 Zookeeper thread leaks in hbase-spark bulkLoad method (Key (tedyu: rev 8143d5afa4a34c5f06a22e30b5017958b8c3f60c) * (edit) hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/HBaseContext.scala > Zookeeper thread leaks in hbase-spark bulkLoad method > - > > Key: HBASE-19848 > URL: https://issues.apache.org/jira/browse/HBASE-19848 > Project: HBase > Issue Type: Bug > Components: spark, Zookeeper >Affects Versions: 1.2.0 > Environment: hbase-spark-1.2.0-cdh5.12.1 version > spark 1.6 >Reporter: Key Hutu >Assignee: Key Hutu >Priority: Major > Labels: performance > Attachments: HBASE-19848-V2.patch, HBASE-19848-V3.patch, > HBaseContext.patch, HBaseContext.scala > > Original Estimate: 72h > Remaining Estimate: 72h > > In hbase-spark project, HBaseContext provides bulkload methond for loading > spark rdd data to hbase easily.But when i using it frequently， the program > will throw "cannot create native thread" exception. > using pstack command in spark driver process , the thread num is increasing > using jstack, named "main-SendThread" and "main-EventThread" thread so many > It seems like that , connection created before bulkload ,but close method > uninvoked at last -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19848) Zookeeper thread leaks in hbase-spark bulkLoad method
[
https://issues.apache.org/jira/browse/HBASE-19848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350222#comment-16350222
]
Hadoop QA commented on HBASE-19848:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} scalac {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
21s{color} | {color:green} hbase-spark in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 27s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19848 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12908972/HBASE-19848-V3.patch |
| Optional Tests | asflicense scalac scaladoc unit compile |
| uname | Linux c6ceb0be5c72 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 2f4d0b94bc |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/11361/testReport/ |
| Max. process+thread count | 1036 (vs. ulimit of 1) |
| modules | C: hbase-spark U: hbase-spark |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/11361/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Zookeeper thread leaks in hbase-spark bulkLoad method
> -
>
> Key: HBASE-19848
> URL: https://issues.apache.org/jira/browse/HBASE-19848
> Project: HBase
> Issue Type: Bug
> Components: spark, Zookeeper
>Affects Versions: 1.2.0
> Environment: hbase-spark-1.2.0-cdh5.12.1 version
> spark 1.6
>Reporter: Key Hutu
>Assignee: Key Hutu
>Priority: Major
> Labels: performance
> Fix For: 1.2.0
>
> Attachments: HBASE-19848-V2.patch, HBASE-19848-V3.patch,
> HBaseContext.patch, HBaseContext.scala
>
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> In hbase-spark project, HBaseContext provides bulkload methond for loading
> spark rdd data to hbase easily.But when i using it frequently， the program
> will throw "cannot create native thread" exception.
> using pstack command in spark driver process , the thread num is increasing
> using jstack, named "main-SendThread" and "main-EventThread" thread so many
> It seems like that , connection created before bulkload ,but close method
> uninvoked at last
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19848) Zookeeper thread leaks in hbase-spark bulkLoad method
[ https://issues.apache.org/jira/browse/HBASE-19848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346842#comment-16346842 ] Ted Yu commented on HBASE-19848: Can you remove the author line ? Please remove whitespace (tab). > Zookeeper thread leaks in hbase-spark bulkLoad method > - > > Key: HBASE-19848 > URL: https://issues.apache.org/jira/browse/HBASE-19848 > Project: HBase > Issue Type: Bug > Components: spark, Zookeeper >Affects Versions: 1.2.0 > Environment: hbase-spark-1.2.0-cdh5.12.1 version > spark 1.6 >Reporter: Key Hutu >Assignee: Key Hutu >Priority: Major > Labels: performance > Fix For: 1.2.0 > > Attachments: HBASE-19848-V2.patch, HBaseContext.patch, > HBaseContext.scala > > Original Estimate: 72h > Remaining Estimate: 72h > > In hbase-spark project, HBaseContext provides bulkload methond for loading > spark rdd data to hbase easily.But when i using it frequently， the program > will throw "cannot create native thread" exception. > using pstack command in spark driver process , the thread num is increasing > using jstack, named "main-SendThread" and "main-EventThread" thread so many > It seems like that , connection created before bulkload ,but close method > uninvoked at last -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19848) Zookeeper thread leaks in hbase-spark bulkLoad method
[
https://issues.apache.org/jira/browse/HBASE-19848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346742#comment-16346742
]
Hadoop QA commented on HBASE-19848:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:red}-1{color} | {color:red} @author {color} | {color:red} 0m
0s{color} | {color:red} The patch appears to contain 1 @author tags which the
community has agreed to not allow in code contributions. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} scalac {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
9s{color} | {color:green} hbase-spark in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19848 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12908560/HBASE-19848-V2.patch |
| Optional Tests | asflicense scalac scaladoc unit compile |
| uname | Linux 415e94c94a1c 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19
15:49:21 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 2e6bc12441 |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| @author |
https://builds.apache.org/job/PreCommit-HBASE-Build/11289/artifact/patchprocess/author-tags.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HBASE-Build/11289/artifact/patchprocess/whitespace-eol.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/11289/testReport/ |
| modules | C: hbase-spark U: hbase-spark |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/11289/console |
| Powered by | Apache Yetus 0.6.0 http://yetus.apache.org |
This message was automatically generated.
> Zookeeper thread leaks in hbase-spark bulkLoad method
> -
>
> Key: HBASE-19848
> URL: https://issues.apache.org/jira/browse/HBASE-19848
> Project: HBase
> Issue Type: Bug
> Components: spark, Zookeeper
>Affects Versions: 1.2.0
> Environment: hbase-spark-1.2.0-cdh5.12.1 version
> spark 1.6
>Reporter: Key Hutu
>Assignee: Key Hutu
>Priority: Major
> Labels: performance
> Fix For: 1.2.0
>
> Attachments: HBASE-19848-V2.patch, HBaseContext.patch,
> HBaseContext.scala
>
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> In hbase-spark project, HBaseContext provides bulkload methond for loading
> spark rdd data to hbase easily.But when i using it frequently， the program
> will throw "cannot create native thread" exception.
> using pstack command in spark driver process , the thread num is increasing
> using jstack, named "main-SendThread" and "main-EventThread" thread so many
[jira] [Commented] (HBASE-19848) Zookeeper thread leaks in hbase-spark bulkLoad method
[ https://issues.apache.org/jira/browse/HBASE-19848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346718#comment-16346718 ] Key Hutu commented on HBASE-19848: -- Hi !Ted Yu, [~huaxiang] I fix compile error locally. In the patch, just make sure the connection is closing, testing the patch in addition no need. What do you think ? Thank you > Zookeeper thread leaks in hbase-spark bulkLoad method > - > > Key: HBASE-19848 > URL: https://issues.apache.org/jira/browse/HBASE-19848 > Project: HBase > Issue Type: Bug > Components: spark, Zookeeper >Affects Versions: 1.2.0 > Environment: hbase-spark-1.2.0-cdh5.12.1 version > spark 1.6 >Reporter: Key Hutu >Assignee: Key Hutu >Priority: Major > Labels: performance > Fix For: 1.2.0 > > Attachments: HBASE-19848-V2.patch, HBaseContext.patch, > HBaseContext.scala > > Original Estimate: 72h > Remaining Estimate: 72h > > In hbase-spark project, HBaseContext provides bulkload methond for loading > spark rdd data to hbase easily.But when i using it frequently， the program > will throw "cannot create native thread" exception. > using pstack command in spark driver process , the thread num is increasing > using jstack, named "main-SendThread" and "main-EventThread" thread so many > It seems like that , connection created before bulkload ,but close method > uninvoked at last -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19848) Zookeeper thread leaks in hbase-spark bulkLoad method
[ https://issues.apache.org/jira/browse/HBASE-19848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346690#comment-16346690 ] Key Hutu commented on HBASE-19848: -- just close the connection, no need to test the patch in addition > Zookeeper thread leaks in hbase-spark bulkLoad method > - > > Key: HBASE-19848 > URL: https://issues.apache.org/jira/browse/HBASE-19848 > Project: HBase > Issue Type: Bug > Components: spark, Zookeeper >Affects Versions: 1.2.0 > Environment: hbase-spark-1.2.0-cdh5.12.1 version > spark 1.6 >Reporter: Key Hutu >Assignee: Key Hutu >Priority: Major > Labels: performance > Fix For: 1.2.0 > > Attachments: HBASE-19848-V2.patch, HBaseContext.patch, > HBaseContext.scala > > Original Estimate: 72h > Remaining Estimate: 72h > > In hbase-spark project, HBaseContext provides bulkload methond for loading > spark rdd data to hbase easily.But when i using it frequently， the program > will throw "cannot create native thread" exception. > using pstack command in spark driver process , the thread num is increasing > using jstack, named "main-SendThread" and "main-EventThread" thread so many > It seems like that , connection created before bulkload ,but close method > uninvoked at last -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19848) Zookeeper thread leaks in hbase-spark bulkLoad method
[
https://issues.apache.org/jira/browse/HBASE-19848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345336#comment-16345336
]
Ted Yu commented on HBASE-19848:
There were several compilation errors, such as:
{code}
[ERROR]
/testptch/hbase/hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/HBaseContext.scala:875:
error: type WriterBuilder is not a member of object
org.apache.hadoop.hbase.regionserver.StoreFile
[ERROR] new StoreFile.WriterBuilder(conf, new CacheConfig(tempConf), new
HFileSystem(fs))
{code}
Please fix, compile locally before uploading the next patch.
Include the JIRA number in the name of patch file.
> Zookeeper thread leaks in hbase-spark bulkLoad method
> -
>
> Key: HBASE-19848
> URL: https://issues.apache.org/jira/browse/HBASE-19848
> Project: HBase
> Issue Type: Bug
> Components: spark, Zookeeper
>Affects Versions: 1.2.0
> Environment: hbase-spark-1.2.0-cdh5.12.1 version
> spark 1.6
>Reporter: Key Hutu
>Assignee: Key Hutu
>Priority: Major
> Labels: performance
> Fix For: 1.2.0
>
> Attachments: HBaseContext.patch, HBaseContext.scala
>
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> In hbase-spark project, HBaseContext provides bulkload methond for loading
> spark rdd data to hbase easily.But when i using it frequently， the program
> will throw "cannot create native thread" exception.
> using pstack command in spark driver process , the thread num is increasing
> using jstack, named "main-SendThread" and "main-EventThread" thread so many
> It seems like that , connection created before bulkload ,but close method
> uninvoked at last
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19848) Zookeeper thread leaks in hbase-spark bulkLoad method
[
https://issues.apache.org/jira/browse/HBASE-19848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345013#comment-16345013
]
Hadoop QA commented on HBASE-19848:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
54s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
26s{color} | {color:red} hbase-spark in the patch failed. {color} |
| {color:red}-1{color} | {color:red} scalac {color} | {color:red} 0m
26s{color} | {color:red} hbase-spark in the patch failed. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} scaladoc {color} | {color:red} 0m
29s{color} | {color:red} hbase-spark generated 9 new + 0 unchanged - 0 fixed =
9 total (was 0) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 26s{color}
| {color:red} hbase-spark in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 9m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19848 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12908326/HBaseContext.patch |
| Optional Tests | asflicense scalac scaladoc unit compile |
| uname | Linux fabae75c577f 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 67f91f1282 |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| compile |
https://builds.apache.org/job/PreCommit-HBASE-Build/11253/artifact/patchprocess/patch-compile-hbase-spark.txt
|
| scalac |
https://builds.apache.org/job/PreCommit-HBASE-Build/11253/artifact/patchprocess/patch-compile-hbase-spark.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HBASE-Build/11253/artifact/patchprocess/whitespace-eol.txt
|
| scaladoc |
https://builds.apache.org/job/PreCommit-HBASE-Build/11253/artifact/patchprocess/diff-scaladoc-scaladoc-hbase-spark.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/11253/artifact/patchprocess/patch-unit-hbase-spark.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/11253/testReport/ |
| Max. process+thread count | 83 (vs. ulimit of 1000) |
| modules | C: hbase-spark U: hbase-spark |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/11253/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Zookeeper thread leaks in hbase-spark bulkLoad method
> -
>
> Key: HBASE-19848
> URL: https://issues.apache.org/jira/browse/HBASE-19848
> Project: HBase
> Issue Type: Bug
> Components: spark, Zookeeper
>Affects Versions: 1.2.0
> Environment: hbase-spark-1.2.0-cdh5.12.1 version
> spark 1.6
>Reporter: Key Hutu
>Assignee: Key Hutu
>Priority: Major
> Labels: performance
> Fix For: 1.2.0
>
> Attachments: HBaseContext.patch, HBaseContext.scala
>
>
[jira] [Commented] (HBASE-19848) Zookeeper thread leaks in hbase-spark bulkLoad method
[ https://issues.apache.org/jira/browse/HBASE-19848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339601#comment-16339601 ] huaxiang sun commented on HBASE-19848: -- Hi [~Key Hutu], as Ted mentioned, you can use the submit-patch script to submit the patch. Or use git diff to create a patch by "git diff --no-prefix". [~yuzhih...@gmail.com] and [~saint@gmail.com], can you add [~Key Hutu] to contributor list so he can assign the Jira to himself? Thanks. > Zookeeper thread leaks in hbase-spark bulkLoad method > - > > Key: HBASE-19848 > URL: https://issues.apache.org/jira/browse/HBASE-19848 > Project: HBase > Issue Type: Bug > Components: spark, Zookeeper >Affects Versions: 1.2.0 > Environment: hbase-spark-1.2.0-cdh5.12.1 version > spark 1.6 >Reporter: Key Hutu >Priority: Major > Labels: performance > Fix For: 1.2.0 > > Attachments: HBaseContext.scala > > Original Estimate: 72h > Remaining Estimate: 72h > > In hbase-spark project, HBaseContext provides bulkload methond for loading > spark rdd data to hbase easily.But when i using it frequently， the program > will throw "cannot create native thread" exception. > using pstack command in spark driver process , the thread num is increasing > using jstack, named "main-SendThread" and "main-EventThread" thread so many > It seems like that , connection created before bulkload ,but close method > uninvoked at last -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19848) Zookeeper thread leaks in hbase-spark bulkLoad method
[ https://issues.apache.org/jira/browse/HBASE-19848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339412#comment-16339412 ] Ted Yu commented on HBASE-19848: You can use dev-support/submit-patch.py to submit patch to this JIRA. QA bot wouldn't accept the java file for test run. > Zookeeper thread leaks in hbase-spark bulkLoad method > - > > Key: HBASE-19848 > URL: https://issues.apache.org/jira/browse/HBASE-19848 > Project: HBase > Issue Type: Bug > Components: spark, Zookeeper >Affects Versions: 1.2.0 > Environment: hbase-spark-1.2.0-cdh5.12.1 version > spark 1.6 >Reporter: Key Hutu >Priority: Major > Labels: performance > Fix For: 1.2.0 > > Attachments: HBaseContext.scala > > Original Estimate: 72h > Remaining Estimate: 72h > > In hbase-spark project, HBaseContext provides bulkload methond for loading > spark rdd data to hbase easily.But when i using it frequently， the program > will throw "cannot create native thread" exception. > using pstack command in spark driver process , the thread num is increasing > using jstack, named "main-SendThread" and "main-EventThread" thread so many > It seems like that , connection created before bulkload ,but close method > uninvoked at last -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19848) Zookeeper thread leaks in hbase-spark bulkLoad method
[ https://issues.apache.org/jira/browse/HBASE-19848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336261#comment-16336261 ] huaxiang sun commented on HBASE-19848: -- Hi [~Key Hutu], can you upload a patch? Thanks. > Zookeeper thread leaks in hbase-spark bulkLoad method > - > > Key: HBASE-19848 > URL: https://issues.apache.org/jira/browse/HBASE-19848 > Project: HBase > Issue Type: Bug > Components: spark, Zookeeper >Affects Versions: 1.2.0 > Environment: hbase-spark-1.2.0-cdh5.12.1 version > spark 1.6 >Reporter: Key Hutu >Priority: Major > Labels: performance > Fix For: 1.2.0 > > Original Estimate: 72h > Remaining Estimate: 72h > > In hbase-spark project, HBaseContext provides bulkload methond for loading > spark rdd data to hbase easily.But when i using it frequently， the program > will throw "cannot create native thread" exception. > using pstack command in spark driver process , the thread num is increasing > using jstack, named "main-SendThread" and "main-EventThread" thread so many > It seems like that , connection created before bulkload ,but close method > uninvoked at last -- This message was sent by Atlassian JIRA (v7.6.3#76005)
