[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15808216#comment-15808216 ] Saikat Kanjilal commented on SPARK-9487: [~srowen] build/tests has passed in jenkins, what do you

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15807969#comment-15807969 ] Apache Spark commented on SPARK-9487: - User 'skanjila' has created a pull request for this issue:

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15739432#comment-15739432 ] Sean Owen commented on SPARK-9487: -- I think this is going around in circles. You already have an open

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15738477#comment-15738477 ] Saikat Kanjilal commented on SPARK-9487: Then I would suggest keeping it open and focus on a

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15738424#comment-15738424 ] Sean Owen commented on SPARK-9487: -- Well, this JIRA is implicitly about making a test or two more robust

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15738419#comment-15738419 ] Saikat Kanjilal commented on SPARK-9487: I'm ok closing it actually but it does outline issues

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15738383#comment-15738383 ] Sean Owen commented on SPARK-9487: -- Yes, that probably means the test changes aren't quite robust in

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15738309#comment-15738309 ] Saikat Kanjilal commented on SPARK-9487: [~srowen] I think the above plan is great minus one

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733832#comment-15733832 ] Sean Owen commented on SPARK-9487: -- (Which thread?) I think that if you can get all tests in one language

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733342#comment-15733342 ] Saikat Kanjilal commented on SPARK-9487: Given the latest thread on the devlist thoughts

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15687208#comment-15687208 ] Saikat Kanjilal commented on SPARK-9487: I really want to finish the effort that I started for

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15686522#comment-15686522 ] Sean Owen commented on SPARK-9487: -- Look at its superclass (trait) that defines "sc", which sets

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15684758#comment-15684758 ] Saikat Kanjilal commented on SPARK-9487: [~srowen] following up, thoughts on how to proceed on

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15680080#comment-15680080 ] Saikat Kanjilal commented on SPARK-9487: Ok guess I spoke too soon :), onto the next set of

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15679880#comment-15679880 ] Saikat Kanjilal commented on SPARK-9487: Ok fixed the unit test, didnt have to resort to using

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674579#comment-15674579 ] Sean Owen commented on SPARK-9487: -- Agree, it seems like it should not be sensitive to ordering within

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674543#comment-15674543 ] Saikat Kanjilal commented on SPARK-9487: Sean, I took a look at the code and here it is:

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15667859#comment-15667859 ] Sean Owen commented on SPARK-9487: -- It may be a slightly flaky test and your environment exposes that in

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15665370#comment-15665370 ] Saikat Kanjilal commented on SPARK-9487: I am running the tests by the following command:

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15663376#comment-15663376 ] Sean Owen commented on SPARK-9487: -- How are you running tests? I'd try to run the tests exactly like

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15660237#comment-15660237 ] Saikat Kanjilal commented on SPARK-9487: Understood, so I wanted a fresh look at this from a

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15659318#comment-15659318 ] Sean Owen commented on SPARK-9487: -- No, it's almost certain that your changes introduced the test

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15658479#comment-15658479 ] Saikat Kanjilal commented on SPARK-9487: ok so I've spent the last hour or so doing deeper

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15658271#comment-15658271 ] Sean Owen commented on SPARK-9487: -- Because the same tests have failed twice, I suspect it is a real

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15658144#comment-15658144 ] Saikat Kanjilal commented on SPARK-9487: No they don't which is why I asked, I will dig into these

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15658073#comment-15658073 ] Sean Owen commented on SPARK-9487: -- All tests are run. Some of the test failures do look related. Do they

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15658055#comment-15658055 ] Saikat Kanjilal commented on SPARK-9487: [~srowen] given this is my first patch, I wanted to

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15657646#comment-15657646 ] Saikat Kanjilal commented on SPARK-9487: [~srowen] if there's no further issues I will : 1) start

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15655682#comment-15655682 ] Apache Spark commented on SPARK-9487: - User 'skanjila' has created a pull request for this issue:

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15651806#comment-15651806 ] Saikat Kanjilal commented on SPARK-9487: Ok , for some odd reason my local branch had the changes

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15651567#comment-15651567 ] Sean Owen commented on SPARK-9487: -- Don't think so ... I searched now and the first 2 hits even weren't

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15651559#comment-15651559 ] Saikat Kanjilal commented on SPARK-9487: Sorry forgot to reply to your other question, from my

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15651556#comment-15651556 ] Sean Owen commented on SPARK-9487: -- I don't see a hurry since the tests aren't broken now. The PR doesn't

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15651553#comment-15651553 ] Saikat Kanjilal commented on SPARK-9487: I definitely want to figure out these test failures as a

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15651525#comment-15651525 ] Sean Owen commented on SPARK-9487: -- I'm not against just committing the Scala (or Java) changes

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15651496#comment-15651496 ] Saikat Kanjilal commented on SPARK-9487: ok I have moved onto python, I am attaching a log that

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634342#comment-15634342 ] Saikat Kanjilal commented on SPARK-9487: added local[4] to repl, sparksql, streaming, all tests

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15633923#comment-15633923 ] Sean Owen commented on SPARK-9487: -- Yes, keep going, why not? > Use the same num. worker threads in

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15633695#comment-15633695 ] Saikat Kanjilal commented on SPARK-9487: [~srowen], [~holdenk] what are the next steps to drive

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15620657#comment-15620657 ] Saikat Kanjilal commented on SPARK-9487: Added org.apache.spark.mllib unitTest changes to pull

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15620613#comment-15620613 ] Saikat Kanjilal commented on SPARK-9487: [~srowen] Yes I read through that and adjusted the PR

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15620610#comment-15620610 ] Apache Spark commented on SPARK-9487: - User 'skanjila' has created a pull request for this issue:

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15620402#comment-15620402 ] Sean Owen commented on SPARK-9487: -- Again have a look at

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15620392#comment-15620392 ] Saikat Kanjilal commented on SPARK-9487: PR attached here:

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15601355#comment-15601355 ] Sean Owen commented on SPARK-9487: -- See

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15599904#comment-15599904 ] Saikat Kanjilal commented on SPARK-9487: Ping on this, [~holdenk] can you let me know if I can

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15587627#comment-15587627 ] Saikat Kanjilal commented on SPARK-9487: [~holdenk] finally getting time to look at this, so I am

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15579847#comment-15579847 ] holdenk commented on SPARK-9487: Great, thanks for taking this issue on :) > Use the same num. worker

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578723#comment-15578723 ] Saikat Kanjilal commented on SPARK-9487: Synched the code, am familiarizing myself first with how

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573380#comment-15573380 ] holdenk commented on SPARK-9487: +1 to [~srowen]'s comment. I would not be surprised to see some test

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15571400#comment-15571400 ] Sean Owen commented on SPARK-9487: -- I think you can just make the change and see what happens? > Use the

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570122#comment-15570122 ] Saikat Kanjilal commented on SPARK-9487: Hello All, Can I help with this in anyway? Thanks > Use

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557018#comment-15557018 ] holdenk commented on SPARK-9487: This will maybe break some tests in the process but it would probably be

[jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests

[ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14941961#comment-14941961 ] Evan Chen commented on SPARK-9487: -- Hey Xiangrui, What would be the preferred num. worker threads?