[jira] [Commented] (SPARK-3630) Identify cause of Kryo+Snappy PARSING_ERROR

2014-10-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14172926#comment-14172926 ] Josh Rosen commented on SPARK-3630: --- I think that there may be multiple causes of these

[jira] [Updated] (SPARK-3630) Identify cause of Kryo+Snappy PARSING_ERROR

2014-10-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-3630: -- Target Version/s: 1.1.1, 1.2.0 Affects Version/s: 1.2.0 Assignee: Josh Rosen > Identif

[jira] [Commented] (SPARK-3958) Possible stream-corruption issues in TorrentBroadcast

2014-10-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14172949#comment-14172949 ] Josh Rosen commented on SPARK-3958: --- Digging into this stacktrace in more detail: Snapp

[jira] [Commented] (SPARK-3937) Unsafe memory access inside of Snappy library

2014-10-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173025#comment-14173025 ] Josh Rosen commented on SPARK-3937: --- Another occurrence of this problem, running a recen

[jira] [Updated] (SPARK-3958) Possible stream-corruption issues in TorrentBroadcast

2014-10-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-3958: -- Target Version/s: 1.2.0 (was: 1.1.1, 1.2.0) > Possible stream-corruption issues in TorrentBroadcast > -

[jira] [Updated] (SPARK-3958) Possible stream-corruption issues in TorrentBroadcast

2014-10-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-3958: -- Affects Version/s: (was: 1.1.0) Removing 1.1.0 as an affected version for now, since the stacktrace

[jira] [Commented] (SPARK-3958) Possible stream-corruption issues in TorrentBroadcast

2014-10-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173154#comment-14173154 ] Josh Rosen commented on SPARK-3958: --- I think that I can safely rule out problems in Torr

[jira] [Commented] (SPARK-3958) Possible stream-corruption issues in TorrentBroadcast

2014-10-15 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173207#comment-14173207 ] Josh Rosen commented on SPARK-3958: --- Hi [~jerryshao], I don't have a reliable reproduct

[jira] [Resolved] (SPARK-2585) Remove special handling of Hadoop JobConf

2014-10-16 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-2585. --- Resolution: Fixed Due to the CONFIGURATION_INSTANTIATION_LOCK thread-safety issue, I think that we'll

[jira] [Commented] (SPARK-3958) Possible stream-corruption issues in TorrentBroadcast

2014-10-16 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174662#comment-14174662 ] Josh Rosen commented on SPARK-3958: --- [~davies] ran across this exception while testing a

[jira] [Resolved] (SPARK-3973) Print callSite information for broadcast variables

2014-10-16 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-3973. --- Resolution: Fixed Issue resolved by pull request 2829 [https://github.com/apache/spark/pull/2829] > P

[jira] [Updated] (SPARK-3985) json file path is not right

2014-10-17 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-3985: -- Affects Version/s: 1.2.0 > json file path is not right > --- > >

[jira] [Resolved] (SPARK-3985) json file path is not right

2014-10-17 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-3985. --- Resolution: Fixed Fix Version/s: 1.2.0 Issue resolved by pull request 2834 [https://github.com/

[jira] [Updated] (SPARK-3926) result of JavaRDD collectAsMap() is not serializable

2014-10-18 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-3926: -- Affects Version/s: 1.2.0 > result of JavaRDD collectAsMap() is not serializable > --

[jira] [Resolved] (SPARK-3926) result of JavaRDD collectAsMap() is not serializable

2014-10-18 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-3926. --- Resolution: Fixed Fix Version/s: 1.2.0 1.1.1 Assignee: Sean Owen Th

[jira] [Updated] (SPARK-3926) result of JavaRDD collectAsMap() is not serializable

2014-10-18 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-3926: -- Affects Version/s: 1.0.2 > result of JavaRDD collectAsMap() is not serializable > --

[jira] [Resolved] (SPARK-534) Make SparkContext thread-safe

2014-10-18 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-534. -- Resolution: Invalid I'm going to resolve this as "invalid" since it's a really old issue and its title /

[jira] [Resolved] (SPARK-3952) Python examples in Streaming Programming Guide

2014-10-18 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-3952. --- Resolution: Fixed Issue resolved by pull request 2808 [https://github.com/apache/spark/pull/2808] > P

[jira] [Resolved] (SPARK-2546) Configuration object thread safety issue

2014-10-19 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-2546. --- Resolution: Fixed Fix Version/s: 1.0.3 1.1.1 1.2.0 Issue

[jira] [Commented] (SPARK-2546) Configuration object thread safety issue

2014-10-19 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14176245#comment-14176245 ] Josh Rosen commented on SPARK-2546: --- I've fixed this in HadoopRDD and applied my fix to

[jira] [Updated] (SPARK-2546) Configuration object thread safety issue

2014-10-19 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-2546: -- Affects Version/s: 1.2.0 1.0.2 1.1.0 > Configuration objec

[jira] [Resolved] (SPARK-3902) Stabilize AsyncRDDActions and expose its methods in Java API

2014-10-19 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-3902. --- Resolution: Fixed Fix Version/s: 1.2.0 Issue resolved by pull request 2760 [https://github.com/

[jira] [Resolved] (SPARK-3948) Sort-based shuffle can lead to assorted stream-corruption exceptions

2014-10-20 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-3948. --- Resolution: Fixed Fix Version/s: 1.1.1 1.2.0 Issue resolved by pull request

[jira] [Commented] (SPARK-3948) Sort-based shuffle can lead to assorted stream-corruption exceptions

2014-10-20 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14177173#comment-14177173 ] Josh Rosen commented on SPARK-3948: --- [~rxin] The patch here should actually fix the issu

[jira] [Resolved] (SPARK-576) Design and develop a more precise progress estimator

2014-10-20 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-576. -- Resolution: Won't Fix Closing this as "Won't Fix"; see our discussion at https://github.com/apache/spark

[jira] [Resolved] (SPARK-3736) Workers should reconnect to Master if disconnected

2014-10-20 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-3736. --- Resolution: Fixed Fix Version/s: 1.2.0 Issue resolved by pull request 2828 [https://github.com/

[jira] [Commented] (SPARK-4019) Repartitioning with more than 2000 partitions drops all data

2014-10-20 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14177617#comment-14177617 ] Josh Rosen commented on SPARK-4019: --- This issue is caused by a bug in HighlyCompressedMa

[jira] [Updated] (SPARK-4019) Repartitioning with more than 2000 partitions may drop all data when partitions are mostly empty.

2014-10-20 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-4019: -- Summary: Repartitioning with more than 2000 partitions may drop all data when partitions are mostly empt

[jira] [Resolved] (SPARK-4015) Documentation in the streaming context references non-existent function

2014-10-21 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-4015. --- Resolution: Fixed Fix Version/s: 1.2.0 Issue resolved by pull request 2861 [https://github.com/

[jira] [Resolved] (SPARK-4035) Wrong format specifier in BlockerManager.scala

2014-10-21 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-4035. --- Resolution: Fixed Fix Version/s: (was: 1.1.1) Issue resolved by pull request 2875 [https://

[jira] [Assigned] (SPARK-3740) Use a compressed bitmap to track zero sized blocks in HighlyCompressedMapStatus

2014-10-21 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen reassigned SPARK-3740: - Assignee: Josh Rosen (was: Liquan Pei) > Use a compressed bitmap to track zero sized blocks in

[jira] [Commented] (SPARK-3740) Use a compressed bitmap to track zero sized blocks in HighlyCompressedMapStatus

2014-10-21 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14178807#comment-14178807 ] Josh Rosen commented on SPARK-3740: --- This isn't just an optimization; it's required for

[jira] [Resolved] (SPARK-3517) mapPartitions is not correct clearing up the closure

2014-10-21 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-3517. --- Resolution: Incomplete Resolving this as "Incomplete" for now, since witgo was unable to reproduce th

[jira] [Assigned] (SPARK-3426) Sort-based shuffle compression behavior is inconsistent

2014-10-21 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen reassigned SPARK-3426: - Assignee: Josh Rosen (was: Andrew Or) > Sort-based shuffle compression behavior is inconsistent

[jira] [Updated] (SPARK-3426) Sort-based shuffle compression behavior is inconsistent

2014-10-21 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-3426: -- Description: We have the following configs: {code} spark.shuffle.compress spark.shuffle.spill.compress

[jira] [Commented] (SPARK-3426) Sort-based shuffle compression behavior is inconsistent

2014-10-21 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14179336#comment-14179336 ] Josh Rosen commented on SPARK-3426: --- I've edited this issue to list the actual exception

[jira] [Commented] (SPARK-3426) Sort-based shuffle compression behavior is inconsistent

2014-10-21 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14179355#comment-14179355 ] Josh Rosen commented on SPARK-3426: --- Based on the discussion in that PR, it sounds folks

[jira] [Updated] (SPARK-3426) Sort-based shuffle compression behavior is inconsistent

2014-10-21 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-3426: -- Affects Version/s: 1.2.0 > Sort-based shuffle compression behavior is inconsistent > ---

[jira] [Updated] (SPARK-3426) Sort-based shuffle compression behavior is inconsistent

2014-10-21 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-3426: -- Description: We have the following configs: {code} spark.shuffle.compress spark.shuffle.spill.compress

[jira] [Created] (SPARK-4044) Thriftserver fails to start when JAVA_HOME points to JRE instead of JDK

2014-10-21 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-4044: - Summary: Thriftserver fails to start when JAVA_HOME points to JRE instead of JDK Key: SPARK-4044 URL: https://issues.apache.org/jira/browse/SPARK-4044 Project: Spark

[jira] [Updated] (SPARK-4006) Spark Driver crashes whenever an Executor is registered twice

2014-10-21 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-4006: -- Description: This is a huge robustness issue for us (Taboola), in mission critical , time sensitive (re

[jira] [Updated] (SPARK-4006) Spark Driver crashes whenever an Executor is registered twice

2014-10-21 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-4006: -- Affects Version/s: 1.2.0 > Spark Driver crashes whenever an Executor is registered twice > -

[jira] [Updated] (SPARK-4006) Spark Driver crashes whenever an Executor is registered twice

2014-10-21 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-4006: -- Target Version/s: 1.1.1, 1.2.0 (was: 1.2.0) > Spark Driver crashes whenever an Executor is registered t

[jira] [Commented] (SPARK-4006) Spark Driver crashes whenever an Executor is registered twice

2014-10-21 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14179625#comment-14179625 ] Josh Rosen commented on SPARK-4006: --- Thanks for the bug report + patch! I'd like to see

[jira] [Created] (SPARK-4049) Storage web UI "fraction cached" shows as > 100%

2014-10-22 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-4049: - Summary: Storage web UI "fraction cached" shows as > 100% Key: SPARK-4049 URL: https://issues.apache.org/jira/browse/SPARK-4049 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-3426) Sort-based shuffle compression behavior is inconsistent

2014-10-22 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-3426. --- Resolution: Fixed Fix Version/s: 1.2.0 1.1.1 Fixed in 1.1.1. and 1.2.0 by my

[jira] [Resolved] (SPARK-3367) Remove spark.shuffle.spill.compress (replace it with existing spark.shuffle.compress)

2014-10-22 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-3367. --- Resolution: Won't Fix Resolving this as "Won't Fix" for now, given the discussion on that PR. We mig

[jira] [Assigned] (SPARK-2353) ArrayIndexOutOfBoundsException in scheduler

2014-10-22 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen reassigned SPARK-2353: - Assignee: Josh Rosen > ArrayIndexOutOfBoundsException in scheduler >

[jira] [Resolved] (SPARK-2353) ArrayIndexOutOfBoundsException in scheduler

2014-10-22 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-2353. --- Resolution: Fixed Fix Version/s: 1.1.0 This looks like a duplicate of SPARK-2931, which was fix

[jira] [Resolved] (SPARK-3709) Executors don't always report broadcast block removal properly back to the driver

2014-10-22 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-3709. --- Resolution: Fixed Fix Version/s: 1.0.3 1.2.0 1.1.1 It loo

[jira] [Updated] (SPARK-4019) Repartitioning with more than 2000 partitions may drop all data when partitions are mostly empty or cause deserialization errors if at least one partition is empty

2014-10-22 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-4019: -- Summary: Repartitioning with more than 2000 partitions may drop all data when partitions are mostly empt

[jira] [Updated] (SPARK-4019) Repartitioning with more than 2000 partitions may drop all data when partitions are mostly empty or cause deserialization errors if at least one partition is empty

2014-10-22 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-4019: -- Description: {code} sc.makeRDD(0 until 10, 1000).repartition(2001).collect() {code} returns `Array()`.

[jira] [Commented] (SPARK-4019) Repartitioning with more than 2000 partitions may drop all data when partitions are mostly empty or cause deserialization errors if at least one partition is empty

2014-10-22 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14180737#comment-14180737 ] Josh Rosen commented on SPARK-4019: --- This also explains another occurrence of the Snappy

[jira] [Commented] (SPARK-3630) Identify cause of Kryo+Snappy PARSING_ERROR

2014-10-22 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14180758#comment-14180758 ] Josh Rosen commented on SPARK-3630: --- I found another cause: *Errors in reduce phases fo

[jira] [Updated] (SPARK-1239) Don't fetch all map output statuses at each reducer during shuffles

2014-10-22 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-1239: -- Assignee: Josh Rosen (was: Kostas Sakellis) I'm re-assigning this to me since I've been working in this

[jira] [Updated] (SPARK-4019) Shuffling with more than 2000 reducers may drop all data when partitions are mostly empty or cause deserialization errors if at least one partition is empty

2014-10-22 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-4019: -- Summary: Shuffling with more than 2000 reducers may drop all data when partitions are mostly empty or ca

[jira] [Updated] (SPARK-4019) Shuffling with more than 2000 map partitions may drop all data when partitions are mostly empty or cause deserialization errors if at least one partition is empty

2014-10-22 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-4019: -- Summary: Shuffling with more than 2000 map partitions may drop all data when partitions are mostly empty

[jira] [Created] (SPARK-4056) Upgrade snappy-java to 1.1.1.4

2014-10-22 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-4056: - Summary: Upgrade snappy-java to 1.1.1.4 Key: SPARK-4056 URL: https://issues.apache.org/jira/browse/SPARK-4056 Project: Spark Issue Type: Improvement Re

[jira] [Commented] (SPARK-3630) Identify cause of Kryo+Snappy PARSING_ERROR

2014-10-22 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14181044#comment-14181044 ] Josh Rosen commented on SPARK-3630: --- snappy-java just published a new release (1.1.1.4)

[jira] [Updated] (SPARK-4019) Shuffling with more than 2000 reducers may drop all data when partitons are mostly empty or cause deserialization errors if at least one partition is empty

2014-10-23 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-4019: -- Summary: Shuffling with more than 2000 reducers may drop all data when partitons are mostly empty or cau

[jira] [Created] (SPARK-4070) Clean up web UI's table rendering code

2014-10-23 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-4070: - Summary: Clean up web UI's table rendering code Key: SPARK-4070 URL: https://issues.apache.org/jira/browse/SPARK-4070 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-3993) python worker may hang after reused from take()

2014-10-23 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-3993. --- Resolution: Fixed Fix Version/s: 1.2.0 Issue resolved by pull request 2838 [https://github.com/

[jira] [Updated] (SPARK-4056) Upgrade snappy-java to 1.1.1.5

2014-10-23 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-4056: -- Description: We should upgrade snappy-java to 1.1.1.5 across all of our maintenance branches. This rel

[jira] [Resolved] (SPARK-4000) Gathers unit tests logs to Jenkins master at the end of a Jenkins build

2014-10-23 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-4000. --- Resolution: Fixed Fix Version/s: 1.2.0 Issue resolved by pull request 2845 [https://github.com/

[jira] [Resolved] (SPARK-4051) Rows in python should support conversion to dictionary

2014-10-24 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-4051. --- Resolution: Fixed Fix Version/s: 1.2.0 Issue resolved by pull request 2896 [https://github.com/

[jira] [Created] (SPARK-4080) "IOException: unexpected exception type" while deserializing tasks

2014-10-24 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-4080: - Summary: "IOException: unexpected exception type" while deserializing tasks Key: SPARK-4080 URL: https://issues.apache.org/jira/browse/SPARK-4080 Project: Spark I

[jira] [Resolved] (SPARK-4080) "IOException: unexpected exception type" while deserializing tasks

2014-10-24 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-4080. --- Resolution: Fixed Fix Version/s: 1.1.1 1.2.0 Issue resolved by pull request

[jira] [Resolved] (SPARK-4056) Upgrade snappy-java to 1.1.1.5

2014-10-24 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-4056. --- Resolution: Fixed Fix Version/s: 1.1.1 1.2.0 Issue resolved by pull request

[jira] [Updated] (SPARK-3789) Python bindings for GraphX

2014-10-24 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-3789: -- Assignee: Kushal Datta > Python bindings for GraphX > -- > > Key

[jira] [Commented] (SPARK-4030) `destroy` method in Broadcast should be public

2014-10-24 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183874#comment-14183874 ] Josh Rosen commented on SPARK-4030: --- This is similar in spirit to SPARK-3885, which is a

[jira] [Resolved] (SPARK-2321) Design a proper progress reporting & event listener API

2014-10-25 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-2321. --- Resolution: Fixed Fix Version/s: 1.2.0 Issue resolved by pull request 2696 [https://github.com/

[jira] [Commented] (SPARK-2321) Design a proper progress reporting & event listener API

2014-10-25 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184010#comment-14184010 ] Josh Rosen commented on SPARK-2321: --- I've merged my pull-based progress-reporting PR, wh

[jira] [Updated] (SPARK-611) Allow JStack to be run from web UI

2014-10-25 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-611: - Component/s: Web UI Assignee: Josh Rosen (was: Patrick Cogan) Patrick and I discussed this yesterda

[jira] [Resolved] (SPARK-4088) Python worker should exit after socket is closed by JVM

2014-10-25 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-4088. --- Resolution: Fixed Fix Version/s: 1.2.0 Issue resolved by pull request 2941 [https://github.com/

[jira] [Resolved] (SPARK-4071) Unroll fails silently if BlockManager size is small

2014-10-25 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-4071. --- Resolution: Fixed Fix Version/s: 1.2.0 Issue resolved by pull request 2917 [https://github.com/

[jira] [Updated] (SPARK-4049) Storage web UI "fraction cached" shows as > 100%

2014-10-26 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-4049: -- Priority: Minor (was: Major) > Storage web UI "fraction cached" shows as > 100% > -

[jira] [Commented] (SPARK-4090) Memory leak in snappy-java 1.1.1.4/5

2014-10-26 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184423#comment-14184423 ] Josh Rosen commented on SPARK-4090: --- I rolled back earlier today, so the build should be

[jira] [Commented] (SPARK-4056) Upgrade snappy-java to 1.1.1.5

2014-10-26 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184570#comment-14184570 ] Josh Rosen commented on SPARK-4056: --- We reverted the 1.1.5 upgrade after discovering tha

[jira] [Commented] (SPARK-4091) Occasionally spark.local.dir can be deleted twice and causes test failure

2014-10-26 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184577#comment-14184577 ] Josh Rosen commented on SPARK-4091: --- This looks like a duplicate of SPARK-3970. > Occas

[jira] [Resolved] (SPARK-1758) failing test org.apache.spark.JavaAPISuite.wholeTextFiles

2014-10-26 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-1758. --- Resolution: Cannot Reproduce Resolving this as "Cannot Reproduce" for now, since I haven't observed th

[jira] [Resolved] (SPARK-3616) Add Selenium tests to Web UI

2014-10-26 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-3616. --- Resolution: Fixed Fix Version/s: 1.2.0 Issue resolved by pull request 2474 [https://github.com/

[jira] [Updated] (SPARK-2698) RDD pages shows negative bytes remaining for some executors

2014-10-26 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-2698: -- Summary: RDD pages shows negative bytes remaining for some executors (was: RDD page Spark Web UI bug)

[jira] [Resolved] (SPARK-2527) incorrect persistence level shown in Spark UI after repersisting

2014-10-26 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-2527. --- Resolution: Cannot Reproduce Assignee: Josh Rosen I think that this was fixed in either 1.1 or 1

[jira] [Updated] (SPARK-2527) incorrect persistence level shown in Spark UI after repersisting

2014-10-26 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-2527: -- Fix Version/s: 1.2.0 > incorrect persistence level shown in Spark UI after repersisting > --

[jira] [Resolved] (SPARK-3021) Job remains in Active Stages after failing

2014-10-26 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-3021. --- Resolution: Cannot Reproduce Fix Version/s: 1.2.0 Assignee: Josh Rosen I tried to repr

[jira] [Commented] (SPARK-2105) SparkUI doesn't remove active stages that failed

2014-10-26 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184613#comment-14184613 ] Josh Rosen commented on SPARK-2105: --- I tried and failed to reproduce this: https://gith

[jira] [Resolved] (SPARK-3274) Spark Streaming Java API reports java.lang.ClassCastException when calling collectAsMap on JavaPairDStream

2014-10-26 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-3274. --- Resolution: Invalid > Spark Streaming Java API reports java.lang.ClassCastException when calling > co

[jira] [Resolved] (SPARK-3590) Expose async APIs in the Java API

2014-10-26 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-3590. --- Resolution: Fixed Fix Version/s: 1.2.0 Assignee: Josh Rosen > Expose async APIs in the

[jira] [Commented] (SPARK-3266) JavaDoubleRDD doesn't contain max()

2014-10-26 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184696#comment-14184696 ] Josh Rosen commented on SPARK-3266: --- I've opened a new pull request which tries to work

[jira] [Resolved] (SPARK-3997) scalastyle should output the error location

2014-10-26 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-3997. --- Resolution: Fixed Fix Version/s: 1.2.0 Issue resolved by pull request 2846 [https://github.com/

[jira] [Created] (SPARK-4105) FAILED_TO_UNCOMPRESS(5) errors when fetching shuffle data with sort-based shuffle

2014-10-27 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-4105: - Summary: FAILED_TO_UNCOMPRESS(5) errors when fetching shuffle data with sort-based shuffle Key: SPARK-4105 URL: https://issues.apache.org/jira/browse/SPARK-4105 Project: Sp

[jira] [Updated] (SPARK-4107) Incorrect handling of Channel.read()'s return value may lead to data truncation

2014-10-27 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-4107: -- Priority: Blocker (was: Major) > Incorrect handling of Channel.read()'s return value may lead to data

[jira] [Created] (SPARK-4107) Incorrect handling of Channel.read()'s return value may lead to data truncation

2014-10-27 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-4107: - Summary: Incorrect handling of Channel.read()'s return value may lead to data truncation Key: SPARK-4107 URL: https://issues.apache.org/jira/browse/SPARK-4107 Project: Spar

[jira] [Commented] (SPARK-4121) Master build failures after shading commons-math3

2014-10-28 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14187458#comment-14187458 ] Josh Rosen commented on SPARK-4121: --- Here's an easy command to reproduce this: {code} m

[jira] [Commented] (SPARK-4133) PARSING_ERROR(2) when upgrading issues from 1.0.2 to 1.1.0

2014-10-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188665#comment-14188665 ] Josh Rosen commented on SPARK-4133: --- Since you mentioned that you see a similar issue wh

[jira] [Updated] (SPARK-3958) Possible stream-corruption issues in TorrentBroadcast

2014-10-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-3958: -- Affects Version/s: 1.1.0 Adding 1.1.0 as an affected version, since a user has observed this in 1.1.0,

[jira] [Commented] (SPARK-4133) PARSING_ERROR(2) when upgrading issues from 1.0.2 to 1.1.0

2014-10-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188815#comment-14188815 ] Josh Rosen commented on SPARK-4133: --- Also, can you paste more of the log leading up to t

[jira] [Commented] (SPARK-3630) Identify cause of Kryo+Snappy PARSING_ERROR

2014-10-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188910#comment-14188910 ] Josh Rosen commented on SPARK-3630: --- *Decompression errors during shuffle fetching*: If

[jira] [Commented] (SPARK-4105) FAILED_TO_UNCOMPRESS(5) errors when fetching shuffle data with sort-based shuffle

2014-10-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188916#comment-14188916 ] Josh Rosen commented on SPARK-4105: --- It seems plausible that SPARK-4107 could have cause

[jira] [Commented] (SPARK-4133) PARSING_ERROR(2) when upgrading issues from 1.0.2 to 1.1.0

2014-10-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14189014#comment-14189014 ] Josh Rosen commented on SPARK-4133: --- Also, could you enable debug logging and share the

<    3   4   5   6   7   8   9   10   11   12   >