[jira] [Commented] (SPARK-3694) Allow printing object graph of tasks/RDD's with a debug flag

[ https://issues.apache.org/jira/browse/SPARK-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14174981#comment-14174981 ] Ilya Ganelin commented on SPARK-3694: - Hello. I would like to work on this. Can you

[jira] [Commented] (SPARK-3694) Allow printing object graph of tasks/RDD's with a debug flag

[ https://issues.apache.org/jira/browse/SPARK-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14176119#comment-14176119 ] Ilya Ganelin commented on SPARK-3694: - Awesome. Thanks Patrick. Allow printing

[jira] [Commented] (SPARK-3080) ArrayIndexOutOfBoundsException in ALS for Large datasets

[ https://issues.apache.org/jira/browse/SPARK-3080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14185846#comment-14185846 ] Ilya Ganelin commented on SPARK-3080: - I've seen the same error on a dataset of ~200

[jira] [Commented] (SPARK-3080) ArrayIndexOutOfBoundsException in ALS for Large datasets

[ https://issues.apache.org/jira/browse/SPARK-3080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14188354#comment-14188354 ] Ilya Ganelin commented on SPARK-3080: - Hello Xiangrui - happy to hear that you're on

[jira] [Commented] (SPARK-3080) ArrayIndexOutOfBoundsException in ALS for Large datasets

[ https://issues.apache.org/jira/browse/SPARK-3080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14189039#comment-14189039 ] Ilya Ganelin commented on SPARK-3080: - Hi all - I have managed to make some

[jira] [Commented] (SPARK-3080) ArrayIndexOutOfBoundsException in ALS for Large datasets

[ https://issues.apache.org/jira/browse/SPARK-3080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14212822#comment-14212822 ] Ilya Ganelin commented on SPARK-3080: - Hi Xiangrui - I was not doing any sort of

[jira] [Commented] (SPARK-3694) Allow printing object graph of tasks/RDD's with a debug flag

[ https://issues.apache.org/jira/browse/SPARK-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14212875#comment-14212875 ] Ilya Ganelin commented on SPARK-3694: - There is also task serialization that happens

[jira] [Commented] (SPARK-3694) Allow printing object graph of tasks/RDD's with a debug flag

[ https://issues.apache.org/jira/browse/SPARK-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14228580#comment-14228580 ] Ilya Ganelin commented on SPARK-3694: - Hi Patrick - I am working on it - I am just

[jira] [Commented] (SPARK-3694) Allow printing object graph of tasks/RDD's with a debug flag

[ https://issues.apache.org/jira/browse/SPARK-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14228631#comment-14228631 ] Ilya Ganelin commented on SPARK-3694: - Tests are completed and I will be submitting a

[jira] [Issue Comment Deleted] (SPARK-4101) [MLLIB] Improve API in Word2Vec model

[ https://issues.apache.org/jira/browse/SPARK-4101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ilya Ganelin updated SPARK-4101: Comment: was deleted (was: If no-one is working on this I would be happy to knock this out. Thanks!

[jira] [Commented] (SPARK-4101) [MLLIB] Improve API in Word2Vec model

[ https://issues.apache.org/jira/browse/SPARK-4101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14229916#comment-14229916 ] Ilya Ganelin commented on SPARK-4101: - Hu Peter - did you have an algorithm in mind

[jira] [Comment Edited] (SPARK-4101) [MLLIB] Improve API in Word2Vec model

[ https://issues.apache.org/jira/browse/SPARK-4101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14229916#comment-14229916 ] Ilya Ganelin edited comment on SPARK-4101 at 12/1/14 3:48 PM: --

[jira] [Commented] (SPARK-4189) FileSegmentManagedBuffer should have a configurable memory map threshold

[ https://issues.apache.org/jira/browse/SPARK-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230579#comment-14230579 ] Ilya Ganelin commented on SPARK-4189: - Looking at the code I see // Just copy the

[jira] [Comment Edited] (SPARK-1962) Add RDD cache reference counting

[ https://issues.apache.org/jira/browse/SPARK-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14230908#comment-14230908 ] Ilya Ganelin edited comment on SPARK-1962 at 12/2/14 3:16 AM: --

[jira] [Commented] (SPARK-4417) New API: sample RDD to fixed number of items

[ https://issues.apache.org/jira/browse/SPARK-4417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14238613#comment-14238613 ] Ilya Ganelin commented on SPARK-4417: - Hi, I'd like to work on this. Can someone

[jira] [Commented] (SPARK-4779) PySpark Shuffle Fails Looking for Files that Don't Exist when low on Memory

[ https://issues.apache.org/jira/browse/SPARK-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14242831#comment-14242831 ] Ilya Ganelin commented on SPARK-4779: - I've seen this issue on Scala as well. This

[jira] [Commented] (SPARK-3533) Add saveAsTextFileByKey() method to RDDs

[ https://issues.apache.org/jira/browse/SPARK-3533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14242913#comment-14242913 ] Ilya Ganelin commented on SPARK-3533: - I am looking into a solution for this. Add

[jira] [Created] (SPARK-4927) Spark does not clean up properly during long jobs.

Ilya Ganelin created SPARK-4927: --- Summary: Spark does not clean up properly during long jobs. Key: SPARK-4927 URL: https://issues.apache.org/jira/browse/SPARK-4927 Project: Spark Issue Type:

[jira] [Commented] (SPARK-4927) Spark does not clean up properly during long jobs.

[ https://issues.apache.org/jira/browse/SPARK-4927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261724#comment-14261724 ] Ilya Ganelin commented on SPARK-4927: - The below code can produce this issue. I've

[jira] [Comment Edited] (SPARK-4927) Spark does not clean up properly during long jobs.

[ https://issues.apache.org/jira/browse/SPARK-4927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261724#comment-14261724 ] Ilya Ganelin edited comment on SPARK-4927 at 12/31/14 12:33 AM:

[jira] [Comment Edited] (SPARK-4927) Spark does not clean up properly during long jobs.

[ https://issues.apache.org/jira/browse/SPARK-4927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261724#comment-14261724 ] Ilya Ganelin edited comment on SPARK-4927 at 12/31/14 12:32 AM:

[jira] [Comment Edited] (SPARK-4927) Spark does not clean up properly during long jobs.

[ https://issues.apache.org/jira/browse/SPARK-4927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261724#comment-14261724 ] Ilya Ganelin edited comment on SPARK-4927 at 12/31/14 12:33 AM:

[jira] [Comment Edited] (SPARK-4927) Spark does not clean up properly during long jobs.

[ https://issues.apache.org/jira/browse/SPARK-4927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261724#comment-14261724 ] Ilya Ganelin edited comment on SPARK-4927 at 12/31/14 12:33 AM:

[jira] [Issue Comment Deleted] (SPARK-4927) Spark does not clean up properly during long jobs.

[ https://issues.apache.org/jira/browse/SPARK-4927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ilya Ganelin updated SPARK-4927: Comment: was deleted (was: The below code can produce this issue. I've also included some log

[jira] [Commented] (SPARK-4927) Spark does not clean up properly during long jobs.

[ https://issues.apache.org/jira/browse/SPARK-4927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14262313#comment-14262313 ] Ilya Ganelin commented on SPARK-4927: - The below code reproduces the problem. Code

[jira] [Commented] (SPARK-4423) Improve foreach() documentation to avoid confusion between local- and cluster-mode behavior

[ https://issues.apache.org/jira/browse/SPARK-4423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14317334#comment-14317334 ] Ilya Ganelin commented on SPARK-4423: - Hi [~pwendell] and [~joshrosen], how do you

[jira] [Comment Edited] (SPARK-4423) Improve foreach() documentation to avoid confusion between local- and cluster-mode behavior

[ https://issues.apache.org/jira/browse/SPARK-4423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14317334#comment-14317334 ] Ilya Ganelin edited comment on SPARK-4423 at 2/12/15 1:46 AM: --

[jira] [Comment Edited] (SPARK-4423) Improve foreach() documentation to avoid confusion between local- and cluster-mode behavior

[ https://issues.apache.org/jira/browse/SPARK-4423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14317334#comment-14317334 ] Ilya Ganelin edited comment on SPARK-4423 at 2/12/15 1:46 AM: --

[jira] [Comment Edited] (SPARK-4423) Improve foreach() documentation to avoid confusion between local- and cluster-mode behavior

[ https://issues.apache.org/jira/browse/SPARK-4423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14317334#comment-14317334 ] Ilya Ganelin edited comment on SPARK-4423 at 2/12/15 2:39 AM: --

[jira] [Comment Edited] (SPARK-4423) Improve foreach() documentation to avoid confusion between local- and cluster-mode behavior

[ https://issues.apache.org/jira/browse/SPARK-4423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14317334#comment-14317334 ] Ilya Ganelin edited comment on SPARK-4423 at 2/12/15 1:43 AM: --

[jira] [Comment Edited] (SPARK-2584) Do not mutate block storage level on the UI

[ https://issues.apache.org/jira/browse/SPARK-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14273877#comment-14273877 ] Ilya Ganelin edited comment on SPARK-2584 at 1/12/15 7:08 PM: --

[jira] [Commented] (SPARK-2584) Do not mutate block storage level on the UI

[ https://issues.apache.org/jira/browse/SPARK-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14273877#comment-14273877 ] Ilya Ganelin commented on SPARK-2584: - Understood, I was looking at the UI for Spark

[jira] [Commented] (SPARK-2584) Do not mutate block storage level on the UI

[ https://issues.apache.org/jira/browse/SPARK-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14273762#comment-14273762 ] Ilya Ganelin commented on SPARK-2584: - Hi Andrew, question about this. When you say we

[jira] [Comment Edited] (SPARK-2584) Do not mutate block storage level on the UI

[ https://issues.apache.org/jira/browse/SPARK-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14273762#comment-14273762 ] Ilya Ganelin edited comment on SPARK-2584 at 1/12/15 4:47 PM: --

[jira] [Commented] (SPARK-4655) Split Stage into ShuffleMapStage and ResultStage subclasses

[ https://issues.apache.org/jira/browse/SPARK-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14312458#comment-14312458 ] Ilya Ganelin commented on SPARK-4655: - Hi [~joshrosen], I'd be happy to work on this.

[jira] [Commented] (SPARK-4423) Improve foreach() documentation to avoid confusion between local- and cluster-mode behavior

[ https://issues.apache.org/jira/browse/SPARK-4423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14312424#comment-14312424 ] Ilya Ganelin commented on SPARK-4423: - I'll be happy to update this. Thank you.

[jira] [Commented] (SPARK-5570) No docs stating that `new SparkConf().set(spark.driver.memory, ...) will not work

[ https://issues.apache.org/jira/browse/SPARK-5570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14312416#comment-14312416 ] Ilya Ganelin commented on SPARK-5570: - I'll fix this, can you please assign it to me?

[jira] [Commented] (SPARK-5079) Detect failed jobs / batches in Spark Streaming unit tests

[ https://issues.apache.org/jira/browse/SPARK-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14312415#comment-14312415 ] Ilya Ganelin commented on SPARK-5079: - I can work on this - can you please assign it

[jira] [Comment Edited] (SPARK-5570) No docs stating that `new SparkConf().set(spark.driver.memory, ...) will not work

[ https://issues.apache.org/jira/browse/SPARK-5570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14312416#comment-14312416 ] Ilya Ganelin edited comment on SPARK-5570 at 2/9/15 4:27 PM: -

[jira] [Commented] (SPARK-823) spark.default.parallelism's default is inconsistent across scheduler backends

[ https://issues.apache.org/jira/browse/SPARK-823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14312382#comment-14312382 ] Ilya Ganelin commented on SPARK-823: Hi [~joshrosen] I believe the documentation is up

[jira] [Commented] (SPARK-3533) Add saveAsTextFileByKey() method to RDDs

[ https://issues.apache.org/jira/browse/SPARK-3533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265122#comment-14265122 ] Ilya Ganelin commented on SPARK-3533: - Hi all - I have that solution (using

[jira] [Commented] (SPARK-3885) Provide mechanism to remove accumulators once they are no longer used

[ https://issues.apache.org/jira/browse/SPARK-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268079#comment-14268079 ] Ilya Ganelin commented on SPARK-3885: - Hi [~joshrosen], I can knock this one out -

[jira] [Commented] (SPARK-4927) Spark does not clean up properly during long jobs.

[ https://issues.apache.org/jira/browse/SPARK-4927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358958#comment-14358958 ] Ilya Ganelin commented on SPARK-4927: - Hi Sean - I have a code snippet that reproduced

[jira] [Commented] (SPARK-4927) Spark does not clean up properly during long jobs.

[ https://issues.apache.org/jira/browse/SPARK-4927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359294#comment-14359294 ] Ilya Ganelin commented on SPARK-4927: - Are you running over yarn? My theory is that

[jira] [Comment Edited] (SPARK-4927) Spark does not clean up properly during long jobs.

[ https://issues.apache.org/jira/browse/SPARK-4927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358958#comment-14358958 ] Ilya Ganelin edited comment on SPARK-4927 at 3/12/15 6:50 PM: --

[jira] [Updated] (SPARK-6616) IsStopped set to true in before stop() is complete.

[ https://issues.apache.org/jira/browse/SPARK-6616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ilya Ganelin updated SPARK-6616: Description: There are numerous instances throughout the code base of the following: {{code}} if

[jira] [Updated] (SPARK-6616) IsStopped set to true in before stop() is complete.

[ https://issues.apache.org/jira/browse/SPARK-6616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ilya Ganelin updated SPARK-6616: Description: There are numerous instances throughout the code base of the following: {{code}} if

[jira] [Commented] (SPARK-6492) SparkContext.stop() can deadlock when DAGSchedulerEventProcessLoop dies

[ https://issues.apache.org/jira/browse/SPARK-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387499#comment-14387499 ] Ilya Ganelin commented on SPARK-6492: - Would it be reasonable to fix this by adding

[jira] [Comment Edited] (SPARK-6492) SparkContext.stop() can deadlock when DAGSchedulerEventProcessLoop dies

[ https://issues.apache.org/jira/browse/SPARK-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387499#comment-14387499 ] Ilya Ganelin edited comment on SPARK-6492 at 3/30/15 10:03 PM:

[jira] [Updated] (SPARK-6616) IsStopped set to true in before stop() is complete.

[ https://issues.apache.org/jira/browse/SPARK-6616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ilya Ganelin updated SPARK-6616: Description: There are numerous instances throughout the code base of the following: {code} if

[jira] [Updated] (SPARK-6616) IsStopped set to true in before stop() is complete.

[ https://issues.apache.org/jira/browse/SPARK-6616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ilya Ganelin updated SPARK-6616: Description: There are numerous instances throughout the code base of the following: {code} if

[jira] [Updated] (SPARK-6616) IsStopped set to true in before stop() is complete.

[ https://issues.apache.org/jira/browse/SPARK-6616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ilya Ganelin updated SPARK-6616: Description: There are numerous instances throughout the code base of the following: {code} if

[jira] [Updated] (SPARK-6616) IsStopped set to true in before stop() is complete.

[ https://issues.apache.org/jira/browse/SPARK-6616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ilya Ganelin updated SPARK-6616: Description: There are numerous instances throughout the code base of the following: {{code}} if

[jira] [Created] (SPARK-6616) IsStopped set to true in before stop() is complete.

Ilya Ganelin created SPARK-6616: --- Summary: IsStopped set to true in before stop() is complete. Key: SPARK-6616 URL: https://issues.apache.org/jira/browse/SPARK-6616 Project: Spark Issue Type:

[jira] [Commented] (SPARK-3533) Add saveAsTextFileByKey() method to RDDs

[ https://issues.apache.org/jira/browse/SPARK-3533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343387#comment-14343387 ] Ilya Ganelin commented on SPARK-3533: - Hey [~aaronjosephs], please feel free. I'm out

[jira] [Commented] (SPARK-5845) Time to cleanup spilled shuffle files not included in shuffle write time

[ https://issues.apache.org/jira/browse/SPARK-5845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339794#comment-14339794 ] Ilya Ganelin commented on SPARK-5845: - I'm code complete on this, will submit a PR

[jira] [Commented] (SPARK-3533) Add saveAsTextFileByKey() method to RDDs

[ https://issues.apache.org/jira/browse/SPARK-3533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14347216#comment-14347216 ] Ilya Ganelin commented on SPARK-3533: - [~aaronjosephs] - Let me see if that's it.

[jira] [Commented] (SPARK-5845) Time to cleanup intermediate shuffle files not included in shuffle write time

[ https://issues.apache.org/jira/browse/SPARK-5845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14333691#comment-14333691 ] Ilya Ganelin commented on SPARK-5845: - Hi Kay - I can knock this one out. Thanks.

[jira] [Commented] (SPARK-5750) Document that ordering of elements in shuffled partitions is not deterministic across runs

[ https://issues.apache.org/jira/browse/SPARK-5750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14333685#comment-14333685 ] Ilya Ganelin commented on SPARK-5750: - Hi Josh - I can knock this out. Thanks.

[jira] [Commented] (SPARK-5079) Detect failed jobs / batches in Spark Streaming unit tests

[ https://issues.apache.org/jira/browse/SPARK-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14333743#comment-14333743 ] Ilya Ganelin commented on SPARK-5079: - Hi [~joshrosen] - I'm trying to wrap my head

[jira] [Commented] (SPARK-5750) Document that ordering of elements in shuffled partitions is not deterministic across runs

[ https://issues.apache.org/jira/browse/SPARK-5750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14336795#comment-14336795 ] Ilya Ganelin commented on SPARK-5750: - Did you have a particular doc in mind to

[jira] [Commented] (SPARK-5750) Document that ordering of elements in shuffled partitions is not deterministic across runs

[ https://issues.apache.org/jira/browse/SPARK-5750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14337104#comment-14337104 ] Ilya Ganelin commented on SPARK-5750: - I'd be happy to pull those in. Is it fine to

[jira] [Commented] (SPARK-5845) Time to cleanup intermediate shuffle files not included in shuffle write time

[ https://issues.apache.org/jira/browse/SPARK-5845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14335646#comment-14335646 ] Ilya Ganelin commented on SPARK-5845: - If I understand correctly, the file cleanup

[jira] [Comment Edited] (SPARK-5845) Time to cleanup intermediate shuffle files not included in shuffle write time

[ https://issues.apache.org/jira/browse/SPARK-5845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14335646#comment-14335646 ] Ilya Ganelin edited comment on SPARK-5845 at 2/24/15 11:19 PM:

[jira] [Commented] (SPARK-5845) Time to cleanup spilled shuffle files not included in shuffle write time

[ https://issues.apache.org/jira/browse/SPARK-5845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14335751#comment-14335751 ] Ilya Ganelin commented on SPARK-5845: - My mistake - missed your comment about the

[jira] [Commented] (SPARK-5945) Spark should not retry a stage infinitely on a FetchFailedException

[ https://issues.apache.org/jira/browse/SPARK-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14367891#comment-14367891 ] Ilya Ganelin commented on SPARK-5945: - Hi Imran - I'd be happy to tackle this. Could

[jira] [Commented] (SPARK-5932) Use consistent naming for byte properties

[ https://issues.apache.org/jira/browse/SPARK-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14367919#comment-14367919 ] Ilya Ganelin commented on SPARK-5932: - [~andrewor14] - I can take this out. Thanks.

[jira] [Comment Edited] (SPARK-5931) Use consistent naming for time properties

[ https://issues.apache.org/jira/browse/SPARK-5931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14367915#comment-14367915 ] Ilya Ganelin edited comment on SPARK-5931 at 3/18/15 9:09 PM: --

[jira] [Commented] (SPARK-5931) Use consistent naming for time properties

[ https://issues.apache.org/jira/browse/SPARK-5931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14367915#comment-14367915 ] Ilya Ganelin commented on SPARK-5931: - @andrewor - I can take this out. Thanks. Use

[jira] [Commented] (SPARK-6703) Provide a way to discover existing SparkContext's

[ https://issues.apache.org/jira/browse/SPARK-6703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14492909#comment-14492909 ] Ilya Ganelin commented on SPARK-6703: - Patrick - what¹s the time line for the 1.4

[jira] [Commented] (SPARK-1021) sortByKey() launches a cluster job when it shouldn't

[ https://issues.apache.org/jira/browse/SPARK-1021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14511230#comment-14511230 ] Ilya Ganelin commented on SPARK-1021: - I'd be happy to look into this and

[jira] [Commented] (SPARK-5945) Spark should not retry a stage infinitely on a FetchFailedException

[ https://issues.apache.org/jira/browse/SPARK-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14508019#comment-14508019 ] Ilya Ganelin commented on SPARK-5945: - [~kayousterhout] - thanks for the review. If I

[jira] [Comment Edited] (SPARK-6891) ExecutorAllocationManager will request negative number executors

[ https://issues.apache.org/jira/browse/SPARK-6891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14507886#comment-14507886 ] Ilya Ganelin edited comment on SPARK-6891 at 4/22/15 8:59 PM: --

[jira] [Commented] (SPARK-6891) ExecutorAllocationManager will request negative number executors

[ https://issues.apache.org/jira/browse/SPARK-6891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14507886#comment-14507886 ] Ilya Ganelin commented on SPARK-6891: - [~meiyoula] I'm running Spark 1.3 (from the

[jira] [Commented] (SPARK-6891) ExecutorAllocationManager will request negative number executors

[ https://issues.apache.org/jira/browse/SPARK-6891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14502322#comment-14502322 ] Ilya Ganelin commented on SPARK-6891: - [~meiyoula] Any hints on reproducing this aside

[jira] [Comment Edited] (SPARK-4514) SparkContext localProperties does not inherit property updates across thread reuse

[ https://issues.apache.org/jira/browse/SPARK-4514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14511537#comment-14511537 ] Ilya Ganelin edited comment on SPARK-4514 at 4/24/15 7:37 PM: --

[jira] [Commented] (SPARK-4514) SparkContext localProperties does not inherit property updates across thread reuse

[ https://issues.apache.org/jira/browse/SPARK-4514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14511537#comment-14511537 ] Ilya Ganelin commented on SPARK-4514: - [~joshrosen] - given your work on SPARK-6629 is

[jira] [Commented] (SPARK-5945) Spark should not retry a stage infinitely on a FetchFailedException

[ https://issues.apache.org/jira/browse/SPARK-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14510154#comment-14510154 ] Ilya Ganelin commented on SPARK-5945: - So to recap: a) Move failure count tracking

[jira] [Commented] (SPARK-7075) Project Tungsten: Improving Physical Execution and Memory Management

[ https://issues.apache.org/jira/browse/SPARK-7075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14521944#comment-14521944 ] Ilya Ganelin commented on SPARK-7075: - This looks like the result of a large internal

[jira] [Commented] (SPARK-6703) Provide a way to discover existing SparkContext's

[ https://issues.apache.org/jira/browse/SPARK-6703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14491877#comment-14491877 ] Ilya Ganelin commented on SPARK-6703: - Patrick - I can look into this. Thank you.

[jira] [Updated] (SPARK-6932) A Prototype of Parameter Server

[ https://issues.apache.org/jira/browse/SPARK-6932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ilya Ganelin updated SPARK-6932: Labels: (was: kjhghbg) A Prototype of Parameter Server ---

[jira] [Updated] (SPARK-6932) A Prototype of Parameter Server

[ https://issues.apache.org/jira/browse/SPARK-6932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ilya Ganelin updated SPARK-6932: Description: h2. Introduction As specified in

[jira] [Created] (SPARK-6746) Refactor large functions in DAGScheduler to improve readibility

Ilya Ganelin created SPARK-6746: --- Summary: Refactor large functions in DAGScheduler to improve readibility Key: SPARK-6746 URL: https://issues.apache.org/jira/browse/SPARK-6746 Project: Spark

[jira] [Commented] (SPARK-6746) Refactor large functions in DAGScheduler to improve readibility

[ https://issues.apache.org/jira/browse/SPARK-6746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14483561#comment-14483561 ] Ilya Ganelin commented on SPARK-6746: - SPARK-5945 requires updating the logic for

[jira] [Created] (SPARK-6780) Add saveAsTextFileByKey method for PySpark

Ilya Ganelin created SPARK-6780: --- Summary: Add saveAsTextFileByKey method for PySpark Key: SPARK-6780 URL: https://issues.apache.org/jira/browse/SPARK-6780 Project: Spark Issue Type:

[jira] [Commented] (SPARK-6780) Add saveAsTextFileByKey method for PySpark

[ https://issues.apache.org/jira/browse/SPARK-6780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485577#comment-14485577 ] Ilya Ganelin commented on SPARK-6780: - SPARK-3533 defines matching methods for Scala

[jira] [Commented] (SPARK-6780) Add saveAsTextFileByKey method for PySpark

[ https://issues.apache.org/jira/browse/SPARK-6780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485582#comment-14485582 ] Ilya Ganelin commented on SPARK-6780: - This code was my attempt to implement this

[jira] [Commented] (SPARK-6780) Add saveAsTextFileByKey method for PySpark

[ https://issues.apache.org/jira/browse/SPARK-6780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485586#comment-14485586 ] Ilya Ganelin commented on SPARK-6780: - Matching test code: {code}

[jira] [Commented] (SPARK-6839) BlockManager.dataDeserialize leaks resources on user exceptions

[ https://issues.apache.org/jira/browse/SPARK-6839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14490545#comment-14490545 ] Ilya Ganelin commented on SPARK-6839: - Imran - I can knock this out. Thanks!

[jira] [Commented] (SPARK-6839) BlockManager.dataDeserialize leaks resources on user exceptions

[ https://issues.apache.org/jira/browse/SPARK-6839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14490576#comment-14490576 ] Ilya Ganelin commented on SPARK-6839: - The obvious solution won't work. Adding a

[jira] [Comment Edited] (SPARK-6839) BlockManager.dataDeserialize leaks resources on user exceptions

[ https://issues.apache.org/jira/browse/SPARK-6839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14490576#comment-14490576 ] Ilya Ganelin edited comment on SPARK-6839 at 4/11/15 12:07 AM:

[jira] [Comment Edited] (SPARK-6839) BlockManager.dataDeserialize leaks resources on user exceptions

[ https://issues.apache.org/jira/browse/SPARK-6839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14490576#comment-14490576 ] Ilya Ganelin edited comment on SPARK-6839 at 4/11/15 12:09 AM:

[jira] [Commented] (SPARK-8056) Design an easier way to construct schema for both Scala and Python

[ https://issues.apache.org/jira/browse/SPARK-8056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14573874#comment-14573874 ] Ilya Ganelin commented on SPARK-8056: - [~rxin] Are you actively working on this? I

[jira] [Comment Edited] (SPARK-8056) Design an easier way to construct schema for both Scala and Python

[ https://issues.apache.org/jira/browse/SPARK-8056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14573874#comment-14573874 ] Ilya Ganelin edited comment on SPARK-8056 at 6/5/15 12:35 AM: --

[jira] [Closed] (SPARK-6746) Refactor large functions in DAGScheduler to improve readibility

[ https://issues.apache.org/jira/browse/SPARK-6746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ilya Ganelin closed SPARK-6746. --- Resolution: Won't Fix Refactor large functions in DAGScheduler to improve readibility

[jira] [Comment Edited] (SPARK-8056) Design an easier way to construct schema for both Scala and Python

[ https://issues.apache.org/jira/browse/SPARK-8056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14574847#comment-14574847 ] Ilya Ganelin edited comment on SPARK-8056 at 6/5/15 5:18 PM: -

[jira] [Commented] (SPARK-8056) Design an easier way to construct schema for both Scala and Python

[ https://issues.apache.org/jira/browse/SPARK-8056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14574847#comment-14574847 ] Ilya Ganelin commented on SPARK-8056: - [~rxin] Sounds good :). Where would you suggest

[jira] [Comment Edited] (SPARK-8056) Design an easier way to construct schema for both Scala and Python

[ https://issues.apache.org/jira/browse/SPARK-8056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14574847#comment-14574847 ] Ilya Ganelin edited comment on SPARK-8056 at 6/5/15 5:17 PM: -

[jira] [Commented] (SPARK-7894) Graph Union Operator

[ https://issues.apache.org/jira/browse/SPARK-7894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14579283#comment-14579283 ] Ilya Ganelin commented on SPARK-7894: - How is this functionality different from the

[jira] [Comment Edited] (SPARK-7894) Graph Union Operator

[ https://issues.apache.org/jira/browse/SPARK-7894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14579283#comment-14579283 ] Ilya Ganelin edited comment on SPARK-7894 at 6/9/15 5:35 PM: -

  1   2   >