Github user gaborgsomogyi commented on the issue:
https://github.com/apache/spark/pull/19893
@vanzin @squito @srowen @jiangxb1987 @henryr
Big thanks to everybody for the constructive comments, learned a lot from
them.
I'll take a look at further possibilities like the
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19893
Merging to master.
It would be nice to file a separate bug to eventually look at how to do
this on the spark-hive module (or maybe it's just not worth the effort).
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19893
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19893
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86094/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19893
**[Test build #86094 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86094/testReport)**
for PR 19893 at commit
Github user gaborgsomogyi commented on the issue:
https://github.com/apache/spark/pull/19893
Checked and seems like unrelated.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19893
Build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19893
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86093/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19893
**[Test build #86093 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86093/testReport)**
for PR 19893 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19893
**[Test build #86094 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86094/testReport)**
for PR 19893 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19893
**[Test build #86093 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86093/testReport)**
for PR 19893 at commit
Github user gaborgsomogyi commented on the issue:
https://github.com/apache/spark/pull/19893
Thread audit disabled in hive.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19893
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86050/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19893
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19893
**[Test build #86050 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86050/testReport)**
for PR 19893 at commit
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19893
Why not disable the thread audit in the hive module? You added that
functionality already, should be pretty trivial to use it.
---
Github user gaborgsomogyi commented on the issue:
https://github.com/apache/spark/pull/19893
Related hive please see my comment on 11 Dec 2017.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19893
**[Test build #86050 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86050/testReport)**
for PR 19893 at commit
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/19893
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user gaborgsomogyi commented on the issue:
https://github.com/apache/spark/pull/19893
Checked the failure but seems like unrelated.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19893
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86035/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19893
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19893
**[Test build #86035 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86035/testReport)**
for PR 19893 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19893
**[Test build #86035 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86035/testReport)**
for PR 19893 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19893
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19893
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85802/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19893
**[Test build #85802 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85802/testReport)**
for PR 19893 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19893
**[Test build #85802 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85802/testReport)**
for PR 19893 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19893
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19893
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85722/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19893
**[Test build #85722 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85722/testReport)**
for PR 19893 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19893
**[Test build #85722 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85722/testReport)**
for PR 19893 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19893
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19893
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85315/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19893
**[Test build #85315 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85315/testReport)**
for PR 19893 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19893
**[Test build #85315 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85315/testReport)**
for PR 19893 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19893
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19893
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85290/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19893
**[Test build #85290 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85290/testReport)**
for PR 19893 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19893
**[Test build #85290 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85290/testReport)**
for PR 19893 at commit
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19893
ok to test
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user gaborgsomogyi commented on the issue:
https://github.com/apache/spark/pull/19893
gentle ping @jiangxb1987
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user gaborgsomogyi commented on the issue:
https://github.com/apache/spark/pull/19893
@squito I mean another jira, because it needs deeper analysis and
discussion.
---
-
To unsubscribe, e-mail:
Github user squito commented on the issue:
https://github.com/apache/spark/pull/19893
lgtm
@jiangxb1987 are you still looking at this?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user squito commented on the issue:
https://github.com/apache/spark/pull/19893
ok I just took a look at BroadcastExchangeExec, I see what you mean. It
isn't *that* bad, since spark isn't continually creating more instances of
those threads (you're not supposed to have more
Github user squito commented on the issue:
https://github.com/apache/spark/pull/19893
> The last suspicious big group of threads (at least for me) is
broadcast-exchange.* but as I've seen this is not false positive because the
threadpool never stopped. In BroadcastExchangeExec:141 a
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19893
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19893
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84915/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19893
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19893
**[Test build #84915 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84915/testReport)**
for PR 19893 at commit
Github user gaborgsomogyi commented on the issue:
https://github.com/apache/spark/pull/19893
The last suspicious big group of threads (at least for me) is
broadcast-exchange.* but as I've seen this is not false positive because the
threadpool never stopped. In
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19893
**[Test build #84915 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84915/testReport)**
for PR 19893 at commit
Github user gaborgsomogyi commented on the issue:
https://github.com/apache/spark/pull/19893
Seems like the new feature caught some false positives in SQL:
```
= THREAD AUDIT POST ACTION CALLED WITHOUT PRE ACTION IN SUITE
o.a.s.sql.sources.DataSourceAnalysisSuite
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19893
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84907/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19893
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19893
**[Test build #84907 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84907/testReport)**
for PR 19893 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19893
**[Test build #84907 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84907/testReport)**
for PR 19893 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19893
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84879/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19893
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19893
**[Test build #84879 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84879/testReport)**
for PR 19893 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19893
**[Test build #84879 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84879/testReport)**
for PR 19893 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19893
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19893
**[Test build #84869 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84869/testReport)**
for PR 19893 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19893
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84869/
Test FAILed.
---
Github user gaborgsomogyi commented on the issue:
https://github.com/apache/spark/pull/19893
@vanzin I've fixed the problematic tests and added a codepart in the
ThreadAudit to highlight such situations. After the build we can see more.
---
Github user gaborgsomogyi commented on the issue:
https://github.com/apache/spark/pull/19893
@squito thanks for sharing your findings, it's helpful. Yeah, slowly
digging into the deep and finding out what these threads are.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19893
**[Test build #84869 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84869/testReport)**
for PR 19893 at commit
Github user squito commented on the issue:
https://github.com/apache/spark/pull/19893
hi I'm just getting back, thanks for looking at this -- one quick comment
on the discussion about why there needs to be a whitelist at all. I have a
vague memory of looking at this a while back,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19893
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84858/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19893
**[Test build #84858 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84858/testReport)**
for PR 19893 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19893
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19893
**[Test build #84858 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84858/testReport)**
for PR 19893 at commit
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19893
You still have not looked at the hive tests.
```
$ grep -a 'POSSIBLE THREAD LEAK' unit-tests.log | wc -l
61
```
A bunch of those look like false positives.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19893
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84763/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19893
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19893
**[Test build #84763 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84763/testReport)**
for PR 19893 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19893
**[Test build #84763 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84763/testReport)**
for PR 19893 at commit
Github user gaborgsomogyi commented on the issue:
https://github.com/apache/spark/pull/19893
@vanzin I've fixed the SQL test flow and additionally I've made the
implementation less invasive by extracting the logic into a trait.
---
Github user gaborgsomogyi commented on the issue:
https://github.com/apache/spark/pull/19893
@jiangxb1987 feel free to take a look at it. More eyes, more possibilities.
---
-
To unsubscribe, e-mail:
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/19893
Oh, it's great that the shared context don't cause much false positives. My
only concern now is that the number of warnings is scary, could you please give
me one or two days to look into the
Github user gaborgsomogyi commented on the issue:
https://github.com/apache/spark/pull/19893
@jiangxb1987 I don't know whether I understand your concern well but there
is no intention to modify the shared `TestHiveContext ` among suites. It will
remain as it is now. As an additional
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/19893
The `TestHiveContext` is shared among test suites, maybe it's not a good
time to change this for now. Could we create a new class that inherit from
`SparkFunSuite` that examines leaking
Github user gaborgsomogyi commented on the issue:
https://github.com/apache/spark/pull/19893
I've analysed the hive related test flow and found SparkSession and
SQLContext sharing between suites as you mentioned. Here is the execution flow:
1. The first hive test suite
Github user gaborgsomogyi commented on the issue:
https://github.com/apache/spark/pull/19893
Still to come. I'll put hive findings here the same way.
---
-
To unsubscribe, e-mail:
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19893
> With this change I don't see any false positives and missed threads.
That sounds good but it only covers sql. Did you also look at hive as I
mentioned previously?
---
Github user gaborgsomogyi commented on the issue:
https://github.com/apache/spark/pull/19893
As a next step analysed SQL test flow. Here are the steps:
1. SharedSparkSession.beforeAll called which initialise SparkSession and
SQLContext
2. SparkFunSuite.beforeAll creates a
Github user gaborgsomogyi commented on the issue:
https://github.com/apache/spark/pull/19893
> Those try to keep the same session alive for multiple suites
Good point to make this part clear.
As a first step I've taken a look at the code and as I see SparkSession as
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19893
Yeah, I don't think this can be added before figuring out SQL tests.
```
$ grep 'POSSIBLE THREAD LEAK' unit-tests.log | wc -l
158
```
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19893
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19893
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84617/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19893
**[Test build #84617 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84617/testReport)**
for PR 19893 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19893
**[Test build #84617 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84617/testReport)**
for PR 19893 at commit
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19893
> All in all if you have an approach which solves context reuse feel free
to share.
I don't have one, but it feels sub-optimal to add code that will knowingly
trigger false positives in a
Github user gaborgsomogyi commented on the issue:
https://github.com/apache/spark/pull/19893
Yeah, this is fully true. This enhancement is definitely will not solve the
issues once and for all. The problems were hidden till now and we would like to
make a step ahead and make it at
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19893
Also, what happens when you run tests in the sql and hive modules? Those
try to keep the same session alive for multiple suites, so I'd expect a bunch
of threads to trigger this code when they
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19893
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84601/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19893
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19893
**[Test build #84601 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84601/testReport)**
for PR 19893 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19893
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84606/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19893
**[Test build #84606 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84606/testReport)**
for PR 19893 at commit
1 - 100 of 129 matches
Mail list logo
