[jira] [Updated] (SPARK-13268) SQL Timestamp stored as GMT but toString returns GMT-08:00

[ https://issues.apache.org/jira/browse/SPARK-13268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ilya Ganelin updated SPARK-13268: - Description: There is an issue with how timestamps are displayed/converted to Strings in Spark

[jira] [Created] (SPARK-13268) SQL Timestamp stored as GMT but toString returns GMT-08:00

Ilya Ganelin created SPARK-13268: Summary: SQL Timestamp stored as GMT but toString returns GMT-08:00 Key: SPARK-13268 URL: https://issues.apache.org/jira/browse/SPARK-13268 Project: Spark

[jira] [Updated] (SPARK-13268) SQL Timestamp stored as GMT but toString returns GMT-08:00

[ https://issues.apache.org/jira/browse/SPARK-13268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ilya Ganelin updated SPARK-13268: - Description: There is an issue with how timestamps are displayed/converted to Strings in Spark

[jira] [Commented] (SPARK-12488) LDA describeTopics() Generates Invalid Term IDs

[ https://issues.apache.org/jira/browse/SPARK-12488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15073236#comment-15073236 ] Ilya Ganelin commented on SPARK-12488: -- I'll submit a dataset that causes this when I have a moment.

[jira] [Updated] (SPARK-12488) LDA describeTopics() Generates Invalid Term IDs

[ https://issues.apache.org/jira/browse/SPARK-12488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ilya Ganelin updated SPARK-12488: - Description: When running the LDA model, and using the describeTopics function, invalid values

[jira] [Updated] (SPARK-12488) LDA Describe Topics Generates Invalid Term IDs

[ https://issues.apache.org/jira/browse/SPARK-12488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ilya Ganelin updated SPARK-12488: - Description: When running the LDA model, and using the describeTopics function, invalid values

[jira] [Commented] (SPARK-12488) LDA describeTopics() Generates Invalid Term IDs

[ https://issues.apache.org/jira/browse/SPARK-12488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15068761#comment-15068761 ] Ilya Ganelin commented on SPARK-12488: -- @jkbradley Would love your feedback here. Thanks! > LDA

[jira] [Updated] (SPARK-12488) LDA describeTopics() Generates Invalid Term IDs

[ https://issues.apache.org/jira/browse/SPARK-12488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ilya Ganelin updated SPARK-12488: - Summary: LDA describeTopics() Generates Invalid Term IDs (was: LDA Describe Topics Generates

[jira] [Created] (SPARK-12488) LDA Describe Topics Generates Invalid Term IDs

Ilya Ganelin created SPARK-12488: Summary: LDA Describe Topics Generates Invalid Term IDs Key: SPARK-12488 URL: https://issues.apache.org/jira/browse/SPARK-12488 Project: Spark Issue Type:

[jira] [Commented] (SPARK-12488) LDA describeTopics() Generates Invalid Term IDs

[ https://issues.apache.org/jira/browse/SPARK-12488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15068827#comment-15068827 ] Ilya Ganelin commented on SPARK-12488: -- Further investigation identifies the issue as stemming from

[jira] [Comment Edited] (SPARK-12488) LDA describeTopics() Generates Invalid Term IDs

[ https://issues.apache.org/jira/browse/SPARK-12488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15068761#comment-15068761 ] Ilya Ganelin edited comment on SPARK-12488 at 12/22/15 9:32 PM:

[jira] [Commented] (SPARK-8890) Reduce memory consumption for dynamic partition insert

[ https://issues.apache.org/jira/browse/SPARK-8890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14628488#comment-14628488 ] Ilya Ganelin commented on SPARK-8890: - [~rxin] I want to make sure I correctly

[jira] [Comment Edited] (SPARK-8890) Reduce memory consumption for dynamic partition insert

[ https://issues.apache.org/jira/browse/SPARK-8890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14628533#comment-14628533 ] Ilya Ganelin edited comment on SPARK-8890 at 7/15/15 6:46 PM: --

[jira] [Commented] (SPARK-8890) Reduce memory consumption for dynamic partition insert

[ https://issues.apache.org/jira/browse/SPARK-8890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14628533#comment-14628533 ] Ilya Ganelin commented on SPARK-8890: - Once data is sorted, is the number of

[jira] [Comment Edited] (SPARK-8890) Reduce memory consumption for dynamic partition insert

[ https://issues.apache.org/jira/browse/SPARK-8890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14628620#comment-14628620 ] Ilya Ganelin edited comment on SPARK-8890 at 7/15/15 7:55 PM: --

[jira] [Commented] (SPARK-8890) Reduce memory consumption for dynamic partition insert

[ https://issues.apache.org/jira/browse/SPARK-8890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14628620#comment-14628620 ] Ilya Ganelin commented on SPARK-8890: - [~yhuai] That makes sense, thank you. Wouldn't

[jira] [Comment Edited] (SPARK-8890) Reduce memory consumption for dynamic partition insert

[ https://issues.apache.org/jira/browse/SPARK-8890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14628620#comment-14628620 ] Ilya Ganelin edited comment on SPARK-8890 at 7/15/15 7:55 PM: --

[jira] [Comment Edited] (SPARK-8890) Reduce memory consumption for dynamic partition insert

[ https://issues.apache.org/jira/browse/SPARK-8890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14628620#comment-14628620 ] Ilya Ganelin edited comment on SPARK-8890 at 7/15/15 7:57 PM: --

[jira] [Comment Edited] (SPARK-8890) Reduce memory consumption for dynamic partition insert

[ https://issues.apache.org/jira/browse/SPARK-8890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14628620#comment-14628620 ] Ilya Ganelin edited comment on SPARK-8890 at 7/15/15 7:57 PM: --

[jira] [Commented] (SPARK-8907) Speed up path construction in DynamicPartitionWriterContainer.outputWriterForRow

[ https://issues.apache.org/jira/browse/SPARK-8907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14625150#comment-14625150 ] Ilya Ganelin commented on SPARK-8907: - [~rxin] The code for this in master has

[jira] [Commented] (SPARK-8464) Consider separating aggregator and non-aggregator paths in ExternalSorter

[ https://issues.apache.org/jira/browse/SPARK-8464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14608292#comment-14608292 ] Ilya Ganelin commented on SPARK-8464: - Josh - I'd be happy to look into this, I'll

[jira] [Commented] (SPARK-3153) shuffle will run out of space when disks have different free space

[ https://issues.apache.org/jira/browse/SPARK-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14608993#comment-14608993 ] Ilya Ganelin commented on SPARK-3153: - cc [~davies] Believe this issue can be closed

[jira] [Commented] (SPARK-7894) Graph Union Operator

[ https://issues.apache.org/jira/browse/SPARK-7894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14579283#comment-14579283 ] Ilya Ganelin commented on SPARK-7894: - How is this functionality different from the

[jira] [Comment Edited] (SPARK-7894) Graph Union Operator

[ https://issues.apache.org/jira/browse/SPARK-7894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14579283#comment-14579283 ] Ilya Ganelin edited comment on SPARK-7894 at 6/9/15 5:35 PM: -

[jira] [Comment Edited] (SPARK-7894) Graph Union Operator

[ https://issues.apache.org/jira/browse/SPARK-7894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14579283#comment-14579283 ] Ilya Ganelin edited comment on SPARK-7894 at 6/9/15 5:37 PM: -

[jira] [Comment Edited] (SPARK-7894) Graph Union Operator

[ https://issues.apache.org/jira/browse/SPARK-7894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14579283#comment-14579283 ] Ilya Ganelin edited comment on SPARK-7894 at 6/9/15 5:38 PM: -

[jira] [Comment Edited] (SPARK-7894) Graph Union Operator

[ https://issues.apache.org/jira/browse/SPARK-7894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14579283#comment-14579283 ] Ilya Ganelin edited comment on SPARK-7894 at 6/9/15 5:35 PM: -

[jira] [Commented] (SPARK-7996) Deprecate the developer api SparkEnv.actorSystem

[ https://issues.apache.org/jira/browse/SPARK-7996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14579643#comment-14579643 ] Ilya Ganelin commented on SPARK-7996: - This seems to be mutually exclusive with

[jira] [Comment Edited] (SPARK-8056) Design an easier way to construct schema for both Scala and Python

[ https://issues.apache.org/jira/browse/SPARK-8056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14574847#comment-14574847 ] Ilya Ganelin edited comment on SPARK-8056 at 6/5/15 5:18 PM: -

[jira] [Commented] (SPARK-8056) Design an easier way to construct schema for both Scala and Python

[ https://issues.apache.org/jira/browse/SPARK-8056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14574847#comment-14574847 ] Ilya Ganelin commented on SPARK-8056: - [~rxin] Sounds good :). Where would you suggest

[jira] [Comment Edited] (SPARK-8056) Design an easier way to construct schema for both Scala and Python

[ https://issues.apache.org/jira/browse/SPARK-8056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14574847#comment-14574847 ] Ilya Ganelin edited comment on SPARK-8056 at 6/5/15 5:17 PM: -

[jira] [Commented] (SPARK-8056) Design an easier way to construct schema for both Scala and Python

[ https://issues.apache.org/jira/browse/SPARK-8056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14573874#comment-14573874 ] Ilya Ganelin commented on SPARK-8056: - [~rxin] Are you actively working on this? I

[jira] [Comment Edited] (SPARK-8056) Design an easier way to construct schema for both Scala and Python

[ https://issues.apache.org/jira/browse/SPARK-8056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14573874#comment-14573874 ] Ilya Ganelin edited comment on SPARK-8056 at 6/5/15 12:35 AM: --

[jira] [Closed] (SPARK-6746) Refactor large functions in DAGScheduler to improve readibility

[ https://issues.apache.org/jira/browse/SPARK-6746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ilya Ganelin closed SPARK-6746. --- Resolution: Won't Fix Refactor large functions in DAGScheduler to improve readibility

[jira] [Commented] (SPARK-7075) Project Tungsten: Improving Physical Execution and Memory Management

[ https://issues.apache.org/jira/browse/SPARK-7075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14521944#comment-14521944 ] Ilya Ganelin commented on SPARK-7075: - This looks like the result of a large internal

[jira] [Commented] (SPARK-1021) sortByKey() launches a cluster job when it shouldn't

[ https://issues.apache.org/jira/browse/SPARK-1021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14511230#comment-14511230 ] Ilya Ganelin commented on SPARK-1021: - I'd be happy to look into this and

[jira] [Comment Edited] (SPARK-4514) SparkContext localProperties does not inherit property updates across thread reuse

[ https://issues.apache.org/jira/browse/SPARK-4514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14511537#comment-14511537 ] Ilya Ganelin edited comment on SPARK-4514 at 4/24/15 7:37 PM: --

[jira] [Commented] (SPARK-4514) SparkContext localProperties does not inherit property updates across thread reuse

[ https://issues.apache.org/jira/browse/SPARK-4514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14511537#comment-14511537 ] Ilya Ganelin commented on SPARK-4514: - [~joshrosen] - given your work on SPARK-6629 is

[jira] [Commented] (SPARK-5945) Spark should not retry a stage infinitely on a FetchFailedException

[ https://issues.apache.org/jira/browse/SPARK-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14510154#comment-14510154 ] Ilya Ganelin commented on SPARK-5945: - So to recap: a) Move failure count tracking

[jira] [Commented] (SPARK-5945) Spark should not retry a stage infinitely on a FetchFailedException

[ https://issues.apache.org/jira/browse/SPARK-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14508019#comment-14508019 ] Ilya Ganelin commented on SPARK-5945: - [~kayousterhout] - thanks for the review. If I

[jira] [Comment Edited] (SPARK-6891) ExecutorAllocationManager will request negative number executors

[ https://issues.apache.org/jira/browse/SPARK-6891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14507886#comment-14507886 ] Ilya Ganelin edited comment on SPARK-6891 at 4/22/15 8:59 PM: --

[jira] [Commented] (SPARK-6891) ExecutorAllocationManager will request negative number executors

[ https://issues.apache.org/jira/browse/SPARK-6891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14507886#comment-14507886 ] Ilya Ganelin commented on SPARK-6891: - [~meiyoula] I'm running Spark 1.3 (from the

[jira] [Commented] (SPARK-6891) ExecutorAllocationManager will request negative number executors

[ https://issues.apache.org/jira/browse/SPARK-6891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14502322#comment-14502322 ] Ilya Ganelin commented on SPARK-6891: - [~meiyoula] Any hints on reproducing this aside

[jira] [Updated] (SPARK-6932) A Prototype of Parameter Server

[ https://issues.apache.org/jira/browse/SPARK-6932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ilya Ganelin updated SPARK-6932: Labels: (was: kjhghbg) A Prototype of Parameter Server ---

[jira] [Updated] (SPARK-6932) A Prototype of Parameter Server

[ https://issues.apache.org/jira/browse/SPARK-6932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ilya Ganelin updated SPARK-6932: Description: h2. Introduction As specified in

[jira] [Commented] (SPARK-6703) Provide a way to discover existing SparkContext's

[ https://issues.apache.org/jira/browse/SPARK-6703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14492909#comment-14492909 ] Ilya Ganelin commented on SPARK-6703: - Patrick - what¹s the time line for the 1.4

[jira] [Commented] (SPARK-6703) Provide a way to discover existing SparkContext's

[ https://issues.apache.org/jira/browse/SPARK-6703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14491877#comment-14491877 ] Ilya Ganelin commented on SPARK-6703: - Patrick - I can look into this. Thank you.

[jira] [Commented] (SPARK-6839) BlockManager.dataDeserialize leaks resources on user exceptions

[ https://issues.apache.org/jira/browse/SPARK-6839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14490545#comment-14490545 ] Ilya Ganelin commented on SPARK-6839: - Imran - I can knock this out. Thanks!

[jira] [Commented] (SPARK-6839) BlockManager.dataDeserialize leaks resources on user exceptions

[ https://issues.apache.org/jira/browse/SPARK-6839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14490576#comment-14490576 ] Ilya Ganelin commented on SPARK-6839: - The obvious solution won't work. Adding a

[jira] [Comment Edited] (SPARK-6839) BlockManager.dataDeserialize leaks resources on user exceptions

[ https://issues.apache.org/jira/browse/SPARK-6839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14490576#comment-14490576 ] Ilya Ganelin edited comment on SPARK-6839 at 4/11/15 12:07 AM:

[jira] [Comment Edited] (SPARK-6839) BlockManager.dataDeserialize leaks resources on user exceptions

[ https://issues.apache.org/jira/browse/SPARK-6839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14490576#comment-14490576 ] Ilya Ganelin edited comment on SPARK-6839 at 4/11/15 12:09 AM:

[jira] [Created] (SPARK-6780) Add saveAsTextFileByKey method for PySpark

Ilya Ganelin created SPARK-6780: --- Summary: Add saveAsTextFileByKey method for PySpark Key: SPARK-6780 URL: https://issues.apache.org/jira/browse/SPARK-6780 Project: Spark Issue Type:

[jira] [Commented] (SPARK-6780) Add saveAsTextFileByKey method for PySpark

[ https://issues.apache.org/jira/browse/SPARK-6780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485577#comment-14485577 ] Ilya Ganelin commented on SPARK-6780: - SPARK-3533 defines matching methods for Scala

[jira] [Commented] (SPARK-6780) Add saveAsTextFileByKey method for PySpark

[ https://issues.apache.org/jira/browse/SPARK-6780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485582#comment-14485582 ] Ilya Ganelin commented on SPARK-6780: - This code was my attempt to implement this

[jira] [Commented] (SPARK-6780) Add saveAsTextFileByKey method for PySpark

[ https://issues.apache.org/jira/browse/SPARK-6780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485586#comment-14485586 ] Ilya Ganelin commented on SPARK-6780: - Matching test code: {code}

[jira] [Created] (SPARK-6746) Refactor large functions in DAGScheduler to improve readibility

Ilya Ganelin created SPARK-6746: --- Summary: Refactor large functions in DAGScheduler to improve readibility Key: SPARK-6746 URL: https://issues.apache.org/jira/browse/SPARK-6746 Project: Spark

[jira] [Commented] (SPARK-6746) Refactor large functions in DAGScheduler to improve readibility

[ https://issues.apache.org/jira/browse/SPARK-6746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14483561#comment-14483561 ] Ilya Ganelin commented on SPARK-6746: - SPARK-5945 requires updating the logic for

[jira] [Updated] (SPARK-6616) IsStopped set to true in before stop() is complete.

[ https://issues.apache.org/jira/browse/SPARK-6616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ilya Ganelin updated SPARK-6616: Description: There are numerous instances throughout the code base of the following: {{code}} if

[jira] [Updated] (SPARK-6616) IsStopped set to true in before stop() is complete.

[ https://issues.apache.org/jira/browse/SPARK-6616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ilya Ganelin updated SPARK-6616: Description: There are numerous instances throughout the code base of the following: {{code}} if

[jira] [Commented] (SPARK-6492) SparkContext.stop() can deadlock when DAGSchedulerEventProcessLoop dies

[ https://issues.apache.org/jira/browse/SPARK-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387499#comment-14387499 ] Ilya Ganelin commented on SPARK-6492: - Would it be reasonable to fix this by adding

[jira] [Comment Edited] (SPARK-6492) SparkContext.stop() can deadlock when DAGSchedulerEventProcessLoop dies

[ https://issues.apache.org/jira/browse/SPARK-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387499#comment-14387499 ] Ilya Ganelin edited comment on SPARK-6492 at 3/30/15 10:03 PM:

[jira] [Updated] (SPARK-6616) IsStopped set to true in before stop() is complete.

[ https://issues.apache.org/jira/browse/SPARK-6616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ilya Ganelin updated SPARK-6616: Description: There are numerous instances throughout the code base of the following: {code} if

[jira] [Updated] (SPARK-6616) IsStopped set to true in before stop() is complete.

[ https://issues.apache.org/jira/browse/SPARK-6616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ilya Ganelin updated SPARK-6616: Description: There are numerous instances throughout the code base of the following: {code} if

[jira] [Updated] (SPARK-6616) IsStopped set to true in before stop() is complete.

[ https://issues.apache.org/jira/browse/SPARK-6616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ilya Ganelin updated SPARK-6616: Description: There are numerous instances throughout the code base of the following: {code} if

[jira] [Updated] (SPARK-6616) IsStopped set to true in before stop() is complete.

[ https://issues.apache.org/jira/browse/SPARK-6616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ilya Ganelin updated SPARK-6616: Description: There are numerous instances throughout the code base of the following: {{code}} if

[jira] [Created] (SPARK-6616) IsStopped set to true in before stop() is complete.

Ilya Ganelin created SPARK-6616: --- Summary: IsStopped set to true in before stop() is complete. Key: SPARK-6616 URL: https://issues.apache.org/jira/browse/SPARK-6616 Project: Spark Issue Type:

[jira] [Commented] (SPARK-5945) Spark should not retry a stage infinitely on a FetchFailedException

[ https://issues.apache.org/jira/browse/SPARK-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14367891#comment-14367891 ] Ilya Ganelin commented on SPARK-5945: - Hi Imran - I'd be happy to tackle this. Could

[jira] [Commented] (SPARK-5932) Use consistent naming for byte properties

[ https://issues.apache.org/jira/browse/SPARK-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14367919#comment-14367919 ] Ilya Ganelin commented on SPARK-5932: - [~andrewor14] - I can take this out. Thanks.

[jira] [Comment Edited] (SPARK-5931) Use consistent naming for time properties

[ https://issues.apache.org/jira/browse/SPARK-5931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14367915#comment-14367915 ] Ilya Ganelin edited comment on SPARK-5931 at 3/18/15 9:09 PM: --

[jira] [Commented] (SPARK-5931) Use consistent naming for time properties

[ https://issues.apache.org/jira/browse/SPARK-5931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14367915#comment-14367915 ] Ilya Ganelin commented on SPARK-5931: - @andrewor - I can take this out. Thanks. Use

[jira] [Commented] (SPARK-4927) Spark does not clean up properly during long jobs.

[ https://issues.apache.org/jira/browse/SPARK-4927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358958#comment-14358958 ] Ilya Ganelin commented on SPARK-4927: - Hi Sean - I have a code snippet that reproduced

[jira] [Commented] (SPARK-4927) Spark does not clean up properly during long jobs.

[ https://issues.apache.org/jira/browse/SPARK-4927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359294#comment-14359294 ] Ilya Ganelin commented on SPARK-4927: - Are you running over yarn? My theory is that

[jira] [Comment Edited] (SPARK-4927) Spark does not clean up properly during long jobs.

[ https://issues.apache.org/jira/browse/SPARK-4927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358958#comment-14358958 ] Ilya Ganelin edited comment on SPARK-4927 at 3/12/15 6:50 PM: --

[jira] [Commented] (SPARK-3533) Add saveAsTextFileByKey() method to RDDs

[ https://issues.apache.org/jira/browse/SPARK-3533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14347216#comment-14347216 ] Ilya Ganelin commented on SPARK-3533: - [~aaronjosephs] - Let me see if that's it.

[jira] [Commented] (SPARK-3533) Add saveAsTextFileByKey() method to RDDs

[ https://issues.apache.org/jira/browse/SPARK-3533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343387#comment-14343387 ] Ilya Ganelin commented on SPARK-3533: - Hey [~aaronjosephs], please feel free. I'm out

[jira] [Commented] (SPARK-5845) Time to cleanup spilled shuffle files not included in shuffle write time

[ https://issues.apache.org/jira/browse/SPARK-5845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339794#comment-14339794 ] Ilya Ganelin commented on SPARK-5845: - I'm code complete on this, will submit a PR

[jira] [Commented] (SPARK-5750) Document that ordering of elements in shuffled partitions is not deterministic across runs

[ https://issues.apache.org/jira/browse/SPARK-5750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14336795#comment-14336795 ] Ilya Ganelin commented on SPARK-5750: - Did you have a particular doc in mind to

[jira] [Commented] (SPARK-5750) Document that ordering of elements in shuffled partitions is not deterministic across runs

[ https://issues.apache.org/jira/browse/SPARK-5750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14337104#comment-14337104 ] Ilya Ganelin commented on SPARK-5750: - I'd be happy to pull those in. Is it fine to

[jira] [Commented] (SPARK-5845) Time to cleanup intermediate shuffle files not included in shuffle write time

[ https://issues.apache.org/jira/browse/SPARK-5845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14335646#comment-14335646 ] Ilya Ganelin commented on SPARK-5845: - If I understand correctly, the file cleanup

[jira] [Comment Edited] (SPARK-5845) Time to cleanup intermediate shuffle files not included in shuffle write time

[ https://issues.apache.org/jira/browse/SPARK-5845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14335646#comment-14335646 ] Ilya Ganelin edited comment on SPARK-5845 at 2/24/15 11:19 PM:

[jira] [Commented] (SPARK-5845) Time to cleanup spilled shuffle files not included in shuffle write time

[ https://issues.apache.org/jira/browse/SPARK-5845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14335751#comment-14335751 ] Ilya Ganelin commented on SPARK-5845: - My mistake - missed your comment about the

[jira] [Commented] (SPARK-5845) Time to cleanup intermediate shuffle files not included in shuffle write time

[ https://issues.apache.org/jira/browse/SPARK-5845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14333691#comment-14333691 ] Ilya Ganelin commented on SPARK-5845: - Hi Kay - I can knock this one out. Thanks.

[jira] [Commented] (SPARK-5750) Document that ordering of elements in shuffled partitions is not deterministic across runs

[ https://issues.apache.org/jira/browse/SPARK-5750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14333685#comment-14333685 ] Ilya Ganelin commented on SPARK-5750: - Hi Josh - I can knock this out. Thanks.

[jira] [Commented] (SPARK-5079) Detect failed jobs / batches in Spark Streaming unit tests

[ https://issues.apache.org/jira/browse/SPARK-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14333743#comment-14333743 ] Ilya Ganelin commented on SPARK-5079: - Hi [~joshrosen] - I'm trying to wrap my head

[jira] [Commented] (SPARK-4423) Improve foreach() documentation to avoid confusion between local- and cluster-mode behavior

[ https://issues.apache.org/jira/browse/SPARK-4423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14317334#comment-14317334 ] Ilya Ganelin commented on SPARK-4423: - Hi [~pwendell] and [~joshrosen], how do you

[jira] [Comment Edited] (SPARK-4423) Improve foreach() documentation to avoid confusion between local- and cluster-mode behavior

[ https://issues.apache.org/jira/browse/SPARK-4423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14317334#comment-14317334 ] Ilya Ganelin edited comment on SPARK-4423 at 2/12/15 1:46 AM: --

[jira] [Comment Edited] (SPARK-4423) Improve foreach() documentation to avoid confusion between local- and cluster-mode behavior

[ https://issues.apache.org/jira/browse/SPARK-4423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14317334#comment-14317334 ] Ilya Ganelin edited comment on SPARK-4423 at 2/12/15 1:46 AM: --

[jira] [Comment Edited] (SPARK-4423) Improve foreach() documentation to avoid confusion between local- and cluster-mode behavior

[ https://issues.apache.org/jira/browse/SPARK-4423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14317334#comment-14317334 ] Ilya Ganelin edited comment on SPARK-4423 at 2/12/15 2:39 AM: --

[jira] [Comment Edited] (SPARK-4423) Improve foreach() documentation to avoid confusion between local- and cluster-mode behavior

[ https://issues.apache.org/jira/browse/SPARK-4423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14317334#comment-14317334 ] Ilya Ganelin edited comment on SPARK-4423 at 2/12/15 1:43 AM: --

[jira] [Commented] (SPARK-4655) Split Stage into ShuffleMapStage and ResultStage subclasses

[ https://issues.apache.org/jira/browse/SPARK-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14312458#comment-14312458 ] Ilya Ganelin commented on SPARK-4655: - Hi [~joshrosen], I'd be happy to work on this.

[jira] [Commented] (SPARK-4423) Improve foreach() documentation to avoid confusion between local- and cluster-mode behavior

[ https://issues.apache.org/jira/browse/SPARK-4423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14312424#comment-14312424 ] Ilya Ganelin commented on SPARK-4423: - I'll be happy to update this. Thank you.

[jira] [Commented] (SPARK-5570) No docs stating that `new SparkConf().set(spark.driver.memory, ...) will not work

[ https://issues.apache.org/jira/browse/SPARK-5570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14312416#comment-14312416 ] Ilya Ganelin commented on SPARK-5570: - I'll fix this, can you please assign it to me?

[jira] [Commented] (SPARK-5079) Detect failed jobs / batches in Spark Streaming unit tests

[ https://issues.apache.org/jira/browse/SPARK-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14312415#comment-14312415 ] Ilya Ganelin commented on SPARK-5079: - I can work on this - can you please assign it

[jira] [Comment Edited] (SPARK-5570) No docs stating that `new SparkConf().set(spark.driver.memory, ...) will not work

[ https://issues.apache.org/jira/browse/SPARK-5570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14312416#comment-14312416 ] Ilya Ganelin edited comment on SPARK-5570 at 2/9/15 4:27 PM: -

[jira] [Commented] (SPARK-823) spark.default.parallelism's default is inconsistent across scheduler backends

[ https://issues.apache.org/jira/browse/SPARK-823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14312382#comment-14312382 ] Ilya Ganelin commented on SPARK-823: Hi [~joshrosen] I believe the documentation is up

[jira] [Comment Edited] (SPARK-2584) Do not mutate block storage level on the UI

[ https://issues.apache.org/jira/browse/SPARK-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14273877#comment-14273877 ] Ilya Ganelin edited comment on SPARK-2584 at 1/12/15 7:08 PM: --

[jira] [Commented] (SPARK-2584) Do not mutate block storage level on the UI

[ https://issues.apache.org/jira/browse/SPARK-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14273877#comment-14273877 ] Ilya Ganelin commented on SPARK-2584: - Understood, I was looking at the UI for Spark

[jira] [Commented] (SPARK-2584) Do not mutate block storage level on the UI

[ https://issues.apache.org/jira/browse/SPARK-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14273762#comment-14273762 ] Ilya Ganelin commented on SPARK-2584: - Hi Andrew, question about this. When you say we

[jira] [Comment Edited] (SPARK-2584) Do not mutate block storage level on the UI

[ https://issues.apache.org/jira/browse/SPARK-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14273762#comment-14273762 ] Ilya Ganelin edited comment on SPARK-2584 at 1/12/15 4:47 PM: --

[jira] [Commented] (SPARK-3885) Provide mechanism to remove accumulators once they are no longer used

[ https://issues.apache.org/jira/browse/SPARK-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268079#comment-14268079 ] Ilya Ganelin commented on SPARK-3885: - Hi [~joshrosen], I can knock this one out -

  1   2   >