[jira] [Commented] (SPARK-22674) PySpark breaks serialization of namedtuple subclasses

[ https://issues.apache.org/jira/browse/SPARK-22674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17447859#comment-17447859 ] Sergei Lebedev commented on SPARK-22674: Hi [~hyukjin.kwon], very happy to see you working on

[jira] [Commented] (SPARK-22338) namedtuple serialization is inefficient

[ https://issues.apache.org/jira/browse/SPARK-22338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458128#comment-16458128 ] Sergei Lebedev commented on SPARK-22338: [This|https://github.com/apache/spark/pull/21180] PR

[jira] [Commented] (SPARK-22674) PySpark breaks serialization of namedtuple subclasses

[ https://issues.apache.org/jira/browse/SPARK-22674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399488#comment-16399488 ] Sergei Lebedev commented on SPARK-22674: [~jonasamrich] I see your point, but the current

[jira] [Commented] (SPARK-22674) PySpark breaks serialization of namedtuple subclasses

[ https://issues.apache.org/jira/browse/SPARK-22674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396948#comment-16396948 ] Sergei Lebedev commented on SPARK-22674: Just out of curiosity: what is the original motivation

[jira] [Commented] (SPARK-22805) Use aliases for StorageLevel in event logs

[ https://issues.apache.org/jira/browse/SPARK-22805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16294745#comment-16294745 ] Sergei Lebedev commented on SPARK-22805: It does, in a sense that 1.6 history server can no

[jira] [Commented] (SPARK-22805) Use aliases for StorageLevel in event logs

[ https://issues.apache.org/jira/browse/SPARK-22805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16294230#comment-16294230 ] Sergei Lebedev commented on SPARK-22805: It doesn't save much storage if block updates tracking

[jira] [Commented] (SPARK-20970) Deprecate TaskMetrics._updatedBlockStatuses

[ https://issues.apache.org/jira/browse/SPARK-20970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16293965#comment-16293965 ] Sergei Lebedev commented on SPARK-20970: Not only does it use a lot of memory on the driver, but

[jira] [Comment Edited] (SPARK-22805) Use aliases for StorageLevel in event logs

[ https://issues.apache.org/jira/browse/SPARK-22805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16293836#comment-16293836 ] Sergei Lebedev edited comment on SPARK-22805 at 12/16/17 11:17 PM: ---

[jira] [Comment Edited] (SPARK-22805) Use aliases for StorageLevel in event logs

[ https://issues.apache.org/jira/browse/SPARK-22805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16293836#comment-16293836 ] Sergei Lebedev edited comment on SPARK-22805 at 12/16/17 10:54 PM: ---

[jira] [Comment Edited] (SPARK-22805) Use aliases for StorageLevel in event logs

[ https://issues.apache.org/jira/browse/SPARK-22805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16293836#comment-16293836 ] Sergei Lebedev edited comment on SPARK-22805 at 12/16/17 8:37 PM: -- I've

[jira] [Commented] (SPARK-22805) Use aliases for StorageLevel in event logs

[ https://issues.apache.org/jira/browse/SPARK-22805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16293836#comment-16293836 ] Sergei Lebedev commented on SPARK-22805: I've emulated the effect of SPARK-20923 by removing all

[jira] [Commented] (SPARK-22805) Use aliases for StorageLevel in event logs

[ https://issues.apache.org/jira/browse/SPARK-22805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16293369#comment-16293369 ] Sergei Lebedev commented on SPARK-22805: After some investigation, it turns out that the majority

[jira] [Comment Edited] (SPARK-22805) Use aliases for StorageLevel in event logs

[ https://issues.apache.org/jira/browse/SPARK-22805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16292758#comment-16292758 ] Sergei Lebedev edited comment on SPARK-22805 at 12/15/17 9:44 PM: -- I

[jira] [Comment Edited] (SPARK-22805) Use aliases for StorageLevel in event logs

[ https://issues.apache.org/jira/browse/SPARK-22805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16292758#comment-16292758 ] Sergei Lebedev edited comment on SPARK-22805 at 12/15/17 9:43 PM: -- I

[jira] [Comment Edited] (SPARK-22805) Use aliases for StorageLevel in event logs

[ https://issues.apache.org/jira/browse/SPARK-22805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16292848#comment-16292848 ] Sergei Lebedev edited comment on SPARK-22805 at 12/15/17 5:25 PM: --

[jira] [Commented] (SPARK-22805) Use aliases for StorageLevel in event logs

[ https://issues.apache.org/jira/browse/SPARK-22805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16292848#comment-16292848 ] Sergei Lebedev commented on SPARK-22805: Here're results for a single application with 6K

[jira] [Commented] (SPARK-22805) Use aliases for StorageLevel in event logs

[ https://issues.apache.org/jira/browse/SPARK-22805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16292758#comment-16292758 ] Sergei Lebedev commented on SPARK-22805: I have a patch which preserves backward compatibility.

[jira] [Comment Edited] (SPARK-22805) Use aliases for StorageLevel in event logs

[ https://issues.apache.org/jira/browse/SPARK-22805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16292758#comment-16292758 ] Sergei Lebedev edited comment on SPARK-22805 at 12/15/17 4:28 PM: -- I

[jira] [Created] (SPARK-22805) Use aliases for StorageLevel in event logs

Sergei Lebedev created SPARK-22805: -- Summary: Use aliases for StorageLevel in event logs Key: SPARK-22805 URL: https://issues.apache.org/jira/browse/SPARK-22805 Project: Spark Issue Type:

[jira] [Commented] (SPARK-19371) Cannot spread cached partitions evenly across executors

[ https://issues.apache.org/jira/browse/SPARK-19371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16257809#comment-16257809 ] Sergei Lebedev commented on SPARK-19371: > Usually the answer is to force a shuffle [...]

[jira] [Commented] (SPARK-22227) DiskBlockManager.getAllBlocks could fail if called during shuffle

[ https://issues.apache.org/jira/browse/SPARK-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197275#comment-16197275 ] Sergei Lebedev commented on SPARK-7: Sidenote: the trace above is caused by the temporary

[jira] [Created] (SPARK-22227) DiskBlockManager.getAllBlocks could fail if called during shuffle

Sergei Lebedev created SPARK-7: -- Summary: DiskBlockManager.getAllBlocks could fail if called during shuffle Key: SPARK-7 URL: https://issues.apache.org/jira/browse/SPARK-7 Project: Spark

[jira] [Created] (SPARK-22147) BlockId.hashCode allocates a StringBuilder/String on each call

Sergei Lebedev created SPARK-22147: -- Summary: BlockId.hashCode allocates a StringBuilder/String on each call Key: SPARK-22147 URL: https://issues.apache.org/jira/browse/SPARK-22147 Project: Spark

[jira] [Commented] (SPARK-792) PairRDDFunctions should expect Product2 instead of Tuple2

[ https://issues.apache.org/jira/browse/SPARK-792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16176268#comment-16176268 ] Sergei Lebedev commented on SPARK-792: -- It looks like the change in the title was never implemented.

[jira] [Created] (SPARK-22062) BlockManager does not account for memory consumed by remote fetches

Sergei Lebedev created SPARK-22062: -- Summary: BlockManager does not account for memory consumed by remote fetches Key: SPARK-22062 URL: https://issues.apache.org/jira/browse/SPARK-22062 Project:

[jira] [Comment Edited] (SPARK-17901) NettyRpcEndpointRef: Error sending message and Caused by: java.util.ConcurrentModificationException

[ https://issues.apache.org/jira/browse/SPARK-17901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155707#comment-16155707 ] Sergei Lebedev edited comment on SPARK-17901 at 9/6/17 5:09 PM:

[jira] [Comment Edited] (SPARK-17901) NettyRpcEndpointRef: Error sending message and Caused by: java.util.ConcurrentModificationException

[ https://issues.apache.org/jira/browse/SPARK-17901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155707#comment-16155707 ] Sergei Lebedev edited comment on SPARK-17901 at 9/6/17 5:09 PM: [~srowen]

[jira] [Commented] (SPARK-17901) NettyRpcEndpointRef: Error sending message and Caused by: java.util.ConcurrentModificationException

[ https://issues.apache.org/jira/browse/SPARK-17901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155707#comment-16155707 ] Sergei Lebedev commented on SPARK-17901: I think this issue could've been closed by mistake: the

[jira] [Commented] (SPARK-20284) Make SerializationStream and DeserializationStream extend Closeable

[ https://issues.apache.org/jira/browse/SPARK-20284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964463#comment-15964463 ] Sergei Lebedev commented on SPARK-20284: Yes, Scala is a bad citizen in the JVM land and comes

[jira] [Comment Edited] (SPARK-20284) Make SerializationStream and DeserializationStream extend Closeable

[ https://issues.apache.org/jira/browse/SPARK-20284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964257#comment-15964257 ] Sergei Lebedev edited comment on SPARK-20284 at 4/11/17 11:57 AM: -- It

[jira] [Commented] (SPARK-20284) Make SerializationStream and DeserializationStream extend Closeable

[ https://issues.apache.org/jira/browse/SPARK-20284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964257#comment-15964257 ] Sergei Lebedev commented on SPARK-20284: It makes the stream well-behaved for any JVM user, e.g.

[jira] [Created] (SPARK-20284) Make SerializationStream and DeserializationStream extend Closeable

Sergei Lebedev created SPARK-20284: -- Summary: Make SerializationStream and DeserializationStream extend Closeable Key: SPARK-20284 URL: https://issues.apache.org/jira/browse/SPARK-20284 Project:

[jira] [Commented] (SPARK-19353) Support binary I/O in PipedRDD

[ https://issues.apache.org/jira/browse/SPARK-19353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852498#comment-15852498 ] Sergei Lebedev commented on SPARK-19353: For reference: we have a fully backward-compatible

[jira] [Created] (SPARK-19353) Support binary I/O in PipedRDD

Sergei Lebedev created SPARK-19353: -- Summary: Support binary I/O in PipedRDD Key: SPARK-19353 URL: https://issues.apache.org/jira/browse/SPARK-19353 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-19010) Include Kryo exception in case of overflow

Sergei Lebedev created SPARK-19010: -- Summary: Include Kryo exception in case of overflow Key: SPARK-19010 URL: https://issues.apache.org/jira/browse/SPARK-19010 Project: Spark Issue Type: