[jira] [Created] (SPARK-6627) Clean up of shuffle code and interfaces

2015-03-31 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-6627: -- Summary: Clean up of shuffle code and interfaces Key: SPARK-6627 URL: https://issues.apache.org/jira/browse/SPARK-6627 Project: Spark Issue Type:

[jira] [Commented] (SPARK-6561) Add partition support in saveAsParquet

2015-03-27 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14383413#comment-14383413 ] Patrick Wendell commented on SPARK-6561: FYI - I just removed Affects Version's

[jira] [Updated] (SPARK-6561) Add partition support in saveAsParquet

2015-03-27 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6561: --- Affects Version/s: (was: 1.3.1) (was: 1.3.0) Add partition

[jira] [Updated] (SPARK-6544) Problem with Avro and Kryo Serialization

2015-03-27 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6544: --- Fix Version/s: 1.3.1 Problem with Avro and Kryo Serialization

[jira] [Resolved] (SPARK-4073) Parquet+Snappy can cause significant off-heap memory usage

2015-03-27 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-4073. Resolution: Won't Fix I have never seen someone else run into this, so closing as not

[jira] [Resolved] (SPARK-5025) Write a guide for creating well-formed packages for Spark

2015-03-27 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-5025. Resolution: Won't Fix I'm closing this as wont fix. There are now a bunch of community

[jira] [Resolved] (SPARK-1844) Support maven-style dependency resolution in sbt build

2015-03-27 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-1844. Resolution: Won't Fix Closing given the combination of (a) this is not that important and

[jira] [Updated] (SPARK-2709) Add a tool for certifying Spark API compatiblity

2015-03-27 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2709: --- Target Version/s: (was: 1.2.0) Add a tool for certifying Spark API compatiblity

[jira] [Updated] (SPARK-2709) Add a tool for certifying Spark API compatiblity

2015-03-27 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2709: --- Priority: Critical (was: Major) Add a tool for certifying Spark API compatiblity

[jira] [Reopened] (SPARK-2709) Add a tool for certifying Spark API compatiblity

2015-03-27 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell reopened SPARK-2709: This came up in some recent conversations. I actually don't think we ever merged this into

[jira] [Resolved] (SPARK-6405) Spark Kryo buffer should be forced to be max. 2GB

2015-03-26 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-6405. Resolution: Fixed Assignee: Matthew Cheah Spark Kryo buffer should be forced to be

[jira] [Resolved] (SPARK-6549) Spark console logger logs to stderr by default

2015-03-26 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-6549. Resolution: Won't Fix I think this is a wont fix due to compatibility issues. If I'm wrong

[jira] [Updated] (SPARK-6499) pyspark: printSchema command on a dataframe hangs

2015-03-25 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6499: --- Component/s: PySpark pyspark: printSchema command on a dataframe hangs

[jira] [Updated] (SPARK-6520) Kyro serialization broken in the shell

2015-03-25 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6520: --- Component/s: Spark Shell Kyro serialization broken in the shell

[jira] [Commented] (SPARK-6481) Set In Progress when a PR is opened for an issue

2015-03-25 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14380432#comment-14380432 ] Patrick Wendell commented on SPARK-6481: Hey All, One issue here, (I think?)

[jira] [Commented] (SPARK-2331) SparkContext.emptyRDD should return RDD[T] not EmptyRDD[T]

2015-03-23 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14376495#comment-14376495 ] Patrick Wendell commented on SPARK-2331: By the way - [~rxin] recently pointed out

[jira] [Reopened] (SPARK-6122) Upgrade Tachyon dependency to 0.6.0

2015-03-23 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell reopened SPARK-6122: I reverted this because it looks like it was responsible for some testing failures due to the

[jira] [Updated] (SPARK-6449) Driver OOM results in reported application result SUCCESS

2015-03-22 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6449: --- Component/s: (was: Spark Core) YARN Driver OOM results in reported

[jira] [Updated] (SPARK-6456) Spark Sql throwing exception on large partitioned data

2015-03-22 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6456: --- Component/s: (was: Spark Core) Spark Sql throwing exception on large partitioned data

[jira] [Resolved] (SPARK-2858) Default log4j configuration no longer seems to work

2015-03-22 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-2858. Resolution: Invalid This is really old and I don't think it still an issue. I'm just

[jira] [Commented] (SPARK-5863) Improve performance of convertToScala codepath.

2015-03-22 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14375229#comment-14375229 ] Patrick Wendell commented on SPARK-5863: This seems worth potentially fixing in

[jira] [Updated] (SPARK-5863) Improve performance of convertToScala codepath.

2015-03-22 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5863: --- Target Version/s: 1.3.1, 1.4.0 (was: 1.4.0) Improve performance of convertToScala codepath.

[jira] [Updated] (SPARK-4227) Document external shuffle service

2015-03-22 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4227: --- Priority: Critical (was: Major) Document external shuffle service

[jira] [Updated] (SPARK-4227) Document external shuffle service

2015-03-22 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4227: --- Target Version/s: 1.3.1, 1.4.0 (was: 1.3.0, 1.4.0) Document external shuffle service

[jira] [Updated] (SPARK-5863) Improve performance of convertToScala codepath.

2015-03-22 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5863: --- Target Version/s: 1.4.0 (was: 1.3.1, 1.4.0) Improve performance of convertToScala codepath.

[jira] [Updated] (SPARK-6012) Deadlock when asking for partitions from CoalescedRDD on top of a TakeOrdered operator

2015-03-22 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6012: --- Target Version/s: 1.4.0 (was: 1.3.1, 1.4.0) Deadlock when asking for partitions from

[jira] [Updated] (SPARK-6012) Deadlock when asking for partitions from CoalescedRDD on top of a TakeOrdered operator

2015-03-22 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6012: --- Target Version/s: 1.3.1, 1.4.0 (was: 1.4.0) Deadlock when asking for partitions from

[jira] [Commented] (SPARK-5863) Improve performance of convertToScala codepath.

2015-03-22 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14375230#comment-14375230 ] Patrick Wendell commented on SPARK-5863: Ah actually - I see [~marmbrus] was the

[jira] [Updated] (SPARK-4925) Publish Spark SQL hive-thriftserver maven artifact

2015-03-20 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4925: --- Fix Version/s: (was: 1.2.1) (was: 1.3.0) Publish Spark SQL

[jira] [Updated] (SPARK-4925) Publish Spark SQL hive-thriftserver maven artifact

2015-03-20 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4925: --- Priority: Critical (was: Major) Publish Spark SQL hive-thriftserver maven artifact

[jira] [Updated] (SPARK-4925) Publish Spark SQL hive-thriftserver maven artifact

2015-03-20 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4925: --- Affects Version/s: (was: 1.2.0) 1.3.0 1.2.1

[jira] [Updated] (SPARK-4123) Show dependency changes in pull requests

2015-03-20 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4123: --- Summary: Show dependency changes in pull requests (was: Show new dependencies added in pull

[jira] [Reopened] (SPARK-4925) Publish Spark SQL hive-thriftserver maven artifact

2015-03-20 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell reopened SPARK-4925: Thanks for bringing this up. Actually - realized this wasn't fixed by some of the other work

[jira] [Updated] (SPARK-4925) Publish Spark SQL hive-thriftserver maven artifact

2015-03-20 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4925: --- Target Version/s: 1.3.1 Publish Spark SQL hive-thriftserver maven artifact

[jira] [Resolved] (SPARK-6404) Call broadcast() in each interval for spark streaming programs.

2015-03-20 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-6404. Resolution: Invalid I'm closing the issue because broadcast variables are immutable, so you

[jira] [Updated] (SPARK-6403) Launch master as spot instance on EC2

2015-03-20 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6403: --- Fix Version/s: (was: 1.2.1) Launch master as spot instance on EC2

[jira] [Updated] (SPARK-6403) Launch master as spot instance on EC2

2015-03-20 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6403: --- Target Version/s: (was: 1.2.1) Launch master as spot instance on EC2

[jira] [Commented] (SPARK-6401) Unable to load a old API input format in Spark streaming

2015-03-20 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14371699#comment-14371699 ] Patrick Wendell commented on SPARK-6401: If this is a matter of just adding a

[jira] [Updated] (SPARK-6414) Spark driver failed with NPE on job cancelation

2015-03-20 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6414: --- Description: When a job group is cancelled, we scan through all jobs to determine which are

[jira] [Updated] (SPARK-6414) Spark driver failed with NPE on job cancelation

2015-03-20 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6414: --- Affects Version/s: 1.3.0 Spark driver failed with NPE on job cancelation

[jira] [Updated] (SPARK-6414) Spark driver failed with NPE on job cancelation

2015-03-20 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6414: --- Priority: Critical (was: Major) Spark driver failed with NPE on job cancelation

[jira] [Commented] (SPARK-5081) Shuffle write increases

2015-03-20 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14372266#comment-14372266 ] Patrick Wendell commented on SPARK-5081: Hey @cbbetz - the last movement on this

[jira] [Updated] (SPARK-6415) Spark Streaming fail-fast: Stop scheduling jobs when a batch fails, and kills the app

2015-03-19 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6415: --- Component/s: Streaming Spark Streaming fail-fast: Stop scheduling jobs when a batch fails,

[jira] [Updated] (SPARK-6362) Broken pipe error when training a RandomForest on a union of two RDDs

2015-03-16 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6362: --- Component/s: PySpark Broken pipe error when training a RandomForest on a union of two RDDs

[jira] [Commented] (SPARK-5310) Update SQL programming guide for 1.3

2015-03-15 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14362757#comment-14362757 ] Patrick Wendell commented on SPARK-5310: [~lian cheng] and [~marmbrus] this

[jira] [Commented] (SPARK-6313) Fetch File Lock file creation doesnt work when Spark working dir is on a NFS mount

2015-03-15 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14362687#comment-14362687 ] Patrick Wendell commented on SPARK-6313: [~joshrosen] changing default caching

[jira] [Updated] (SPARK-6313) Fetch File Lock file creation doesnt work when Spark working dir is on a NFS mount

2015-03-15 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6313: --- Target Version/s: 1.3.1 Fetch File Lock file creation doesnt work when Spark working dir is

[jira] [Updated] (SPARK-4964) Exactly-once + WAL-free Kafka Support in Spark Streaming

2015-03-13 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4964: --- Assignee: Cody Koeninger Exactly-once + WAL-free Kafka Support in Spark Streaming

[jira] [Updated] (SPARK-6313) Fetch File Lock file creation doesnt work when Spark working dir is on a NFS mount

2015-03-13 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6313: --- Priority: Critical (was: Major) Fetch File Lock file creation doesnt work when Spark

[jira] [Resolved] (SPARK-6311) ChiSqTest should check for too few counts

2015-03-12 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-6311. Resolution: Duplicate ChiSqTest should check for too few counts

[jira] [Resolved] (SPARK-6310) ChiSqTest should check for too few counts

2015-03-12 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-6310. Resolution: Duplicate ChiSqTest should check for too few counts

[jira] [Commented] (SPARK-5654) Integrate SparkR into Apache Spark

2015-03-12 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359166#comment-14359166 ] Patrick Wendell commented on SPARK-5654: I see the decision here as somewhat

[jira] [Resolved] (SPARK-4924) Factor out code to launch Spark applications into a separate library

2015-03-11 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-4924. Resolution: Fixed Fix Version/s: 1.4.0 Glad to finally have this in. Thanks for all

[jira] [Updated] (SPARK-6050) Spark on YARN does not work --executor-cores is specified

2015-03-09 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6050: --- Fix Version/s: (was: 1.4.0) Spark on YARN does not work --executor-cores is specified

[jira] [Commented] (SPARK-5134) Bump default Hadoop version to 2+

2015-03-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14352268#comment-14352268 ] Patrick Wendell commented on SPARK-5134: Hey [~rdub] [~srowen], As part of the

[jira] [Commented] (SPARK-1239) Don't fetch all map output statuses at each reducer during shuffles

2015-03-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14352356#comment-14352356 ] Patrick Wendell commented on SPARK-1239: It would be helpful if any users who have

[jira] [Comment Edited] (SPARK-5134) Bump default Hadoop version to 2+

2015-03-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14352268#comment-14352268 ] Patrick Wendell edited comment on SPARK-5134 at 3/8/15 11:27 PM:

[jira] [Commented] (SPARK-5134) Bump default Hadoop version to 2+

2015-03-08 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14352341#comment-14352341 ] Patrick Wendell commented on SPARK-5134: [~shivaram] did it end up working alright

[jira] [Updated] (SPARK-6189) Pandas to DataFrame conversion should check field names for periods

2015-03-07 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6189: --- Component/s: DataFrame Pandas to DataFrame conversion should check field names for periods

[jira] [Updated] (SPARK-6208) executor-memory does not work when using local cluster

2015-03-07 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6208: --- Issue Type: New Feature (was: Bug) executor-memory does not work when using local cluster

[jira] [Updated] (SPARK-4123) Show new dependencies added in pull requests

2015-03-07 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4123: --- Assignee: Brennon York Show new dependencies added in pull requests

[jira] [Commented] (SPARK-4123) Show new dependencies added in pull requests

2015-03-07 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351932#comment-14351932 ] Patrick Wendell commented on SPARK-4123: Hey [~boyork] sorry for the delay. Are

[jira] [Updated] (SPARK-6154) Build error with Scala 2.11 for v1.3.0-rc2

2015-03-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6154: --- Component/s: (was: SQL) Build Build error with Scala 2.11 for

[jira] [Commented] (SPARK-6154) Build error with Scala 2.11 for v1.3.0-rc2

2015-03-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14350915#comment-14350915 ] Patrick Wendell commented on SPARK-6154: Can you give the exact set of flags you

[jira] [Commented] (SPARK-6154) Build error with Scala 2.11 for v1.3.0-rc2

2015-03-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14350975#comment-14350975 ] Patrick Wendell commented on SPARK-6154: Oh I remember now, we don't support this

[jira] [Updated] (SPARK-5183) Document data source API

2015-03-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5183: --- Priority: Critical (was: Blocker) Document data source API

[jira] [Updated] (SPARK-5310) Update SQL programming guide for 1.3

2015-03-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5310: --- Priority: Critical (was: Blocker) Update SQL programming guide for 1.3

[jira] [Updated] (SPARK-6128) Update Spark Streaming Guide for Spark 1.3

2015-03-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6128: --- Priority: Critical (was: Blocker) Update Spark Streaming Guide for Spark 1.3

[jira] [Updated] (SPARK-5345) Flaky test: o.a.s.deploy.history.FsHistoryProviderSuite

2015-03-06 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5345: --- Fix Version/s: 1.3.0 Flaky test: o.a.s.deploy.history.FsHistoryProviderSuite

[jira] [Resolved] (SPARK-6182) spark-parent pom needs to be published for both 2.10 and 2.11

2015-03-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-6182. Resolution: Fixed Fix Version/s: 1.3.0 Assignee: Sean Owen spark-parent

[jira] [Updated] (SPARK-6175) Executor log links are using internal addresses in EC2; display `:0` when ephemeral ports are used

2015-03-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6175: --- Priority: Blocker (was: Major) Executor log links are using internal addresses in EC2;

[jira] [Updated] (SPARK-6141) Upgrade Breeze to 0.11 to fix convergence bug

2015-03-05 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6141: --- Fix Version/s: (was: 1.3.1) 1.3.0 Upgrade Breeze to 0.11 to fix

[jira] [Updated] (SPARK-6149) Spark SQL CLI doesn't work when compiled against Hive 12 with SBT because of runtime incompatibility issues caused by Guava 15

2015-03-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6149: --- Priority: Critical (was: Blocker) Spark SQL CLI doesn't work when compiled against Hive 12

[jira] [Commented] (SPARK-6149) Spark SQL CLI doesn't work when compiled against Hive 12 with SBT because of runtime incompatibility issues caused by Guava 15

2015-03-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14347210#comment-14347210 ] Patrick Wendell commented on SPARK-6149: Since this only affects the sbt build and

[jira] [Commented] (SPARK-5143) spark-network-yarn 2.11 depends on spark-network-shuffle 2.10

2015-03-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14347567#comment-14347567 ] Patrick Wendell commented on SPARK-5143: Yes - good catch Sean. Curious that this

[jira] [Updated] (SPARK-6144) When in cluster mode using ADD JAR with a hdfs:// sourced jar will fail

2015-03-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6144: --- Component/s: Spark Core When in cluster mode using ADD JAR with a hdfs:// sourced jar will

[jira] [Created] (SPARK-6182) spark-parent pom needs to be published for both 2.10 and 2.11

2015-03-04 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-6182: -- Summary: spark-parent pom needs to be published for both 2.10 and 2.11 Key: SPARK-6182 URL: https://issues.apache.org/jira/browse/SPARK-6182 Project: Spark

[jira] [Resolved] (SPARK-6149) Spark SQL CLI doesn't work when compiled against Hive 12 with SBT because of runtime incompatibility issues caused by Guava 15

2015-03-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-6149. Resolution: Fixed Fix Version/s: 1.3.0 Spark SQL CLI doesn't work when compiled

[jira] [Resolved] (SPARK-5143) spark-network-yarn 2.11 depends on spark-network-shuffle 2.10

2015-03-04 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-5143. Resolution: Fixed Fix Version/s: 1.3.0 spark-network-yarn 2.11 depends on

[jira] [Commented] (SPARK-6149) Spark SQL CLI doesn't work when compiled against Hive 12 with SBT because of runtime incompatibility issues caused by Guava 15

2015-03-03 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346515#comment-14346515 ] Patrick Wendell commented on SPARK-6149: Yes - because of this I think simply

[jira] [Commented] (SPARK-6149) Spark SQL CLI doesn't work when compiled against Hive 12 with SBT because of runtime incompatibility issues caused by Guava 15

2015-03-03 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346519#comment-14346519 ] Patrick Wendell commented on SPARK-6149: To be more specific, I am suggesting that

[jira] [Updated] (SPARK-6144) When in cluster mode using ADD JAR with a hdfs:// sourced jar will fail

2015-03-03 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6144: --- Target Version/s: 1.3.0 When in cluster mode using ADD JAR with a hdfs:// sourced jar will

[jira] [Updated] (SPARK-6144) When in cluster mode using ADD JAR with a hdfs:// sourced jar will fail

2015-03-03 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6144: --- Priority: Blocker (was: Major) When in cluster mode using ADD JAR with a hdfs:// sourced

[jira] [Updated] (SPARK-6122) Upgrade Tachyon dependency to 0.6.0

2015-03-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6122: --- Assignee: Calvin Jia Upgrade Tachyon dependency to 0.6.0

[jira] [Resolved] (SPARK-6048) SparkConf.translateConfKey should not translate on set

2015-03-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-6048. Resolution: Fixed Fix Version/s: 1.3.0 SparkConf.translateConfKey should not

[jira] [Updated] (SPARK-6122) Upgrade Tachyon dependency to 0.6.0

2015-03-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6122: --- Assignee: (was: Patrick Wendell) Upgrade Tachyon dependency to 0.6.0

[jira] [Updated] (SPARK-6122) Upgrade Tachyon dependency to 0.6.0

2015-03-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6122: --- Target Version/s: 1.4.0 Upgrade Tachyon dependency to 0.6.0

[jira] [Updated] (SPARK-6122) Upgrade Tachyon dependency to 0.6.0

2015-03-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6122: --- Fix Version/s: (was: 1.3.0) Upgrade Tachyon dependency to 0.6.0

[jira] [Updated] (SPARK-6122) Upgrade Tachyon dependency to 0.6.0

2015-03-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6122: --- Assignee: Patrick Wendell Upgrade Tachyon dependency to 0.6.0

[jira] [Resolved] (SPARK-6066) Metadata in event log makes it very difficult for external libraries to parse event log

2015-03-02 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-6066. Resolution: Fixed Fix Version/s: 1.3.0 Thanks Andrew and Marcelo for your work on

[jira] [Updated] (SPARK-6087) Provide actionable exception if Kryo buffer is not large enough

2015-03-01 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6087: --- Labels: starter (was: ) Provide actionable exception if Kryo buffer is not large enough

[jira] [Commented] (SPARK-6066) Metadata in event log makes it very difficult for external libraries to parse event log

2015-02-28 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341881#comment-14341881 ] Patrick Wendell commented on SPARK-6066: [~vanzin] - yes you are right (an early

[jira] [Updated] (SPARK-6086) Exceptions in DAGScheduler.updateAccumulators

2015-02-28 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6086: --- Component/s: Spark Core Exceptions in DAGScheduler.updateAccumulators

[jira] [Updated] (SPARK-6086) Exceptions in DAGScheduler.updateAccumulators

2015-02-28 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6086: --- Description: Class Cast Exceptions in DAGScheduler.updateAccumulators, when DAGScheduler is

[jira] [Created] (SPARK-6087) Provide actionable exception if Kryo buffer is not large enough

2015-02-28 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-6087: -- Summary: Provide actionable exception if Kryo buffer is not large enough Key: SPARK-6087 URL: https://issues.apache.org/jira/browse/SPARK-6087 Project: Spark

[jira] [Updated] (SPARK-6087) Provide actionable exception if Kryo buffer is not large enough

2015-02-28 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6087: --- Description: Right now if you don't have a large enough Kryo buffer, you get a really

[jira] [Resolved] (SPARK-5979) `--packages` should not exclude spark streaming assembly jars for kafka and flume

2015-02-27 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-5979. Resolution: Fixed Fix Version/s: 1.3.0 `--packages` should not exclude spark

[jira] [Resolved] (SPARK-6070) Yarn Shuffle Service jar packages too many dependencies

2015-02-27 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-6070. Resolution: Fixed Fix Version/s: 1.3.0 Assignee: Marcelo Vanzin Yarn

[jira] [Updated] (SPARK-5979) `--packages` should not exclude spark streaming assembly jars for kafka and flume

2015-02-27 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-5979: --- Assignee: Burak Yavuz `--packages` should not exclude spark streaming assembly jars for

[jira] [Resolved] (SPARK-6032) Move ivy logging to System.err in --packages

2015-02-27 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-6032. Resolution: Fixed Fix Version/s: 1.3.0 Assignee: Burak Yavuz Move ivy

<    1   2   3   4   5   6   7   8   9   10   >