[jira] [Created] (SPARK-3962) Mark spark dependency as "provided" in external libraries

2014-10-15 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-3962: -- Summary: Mark spark dependency as "provided" in external libraries Key: SPARK-3962 URL: https://issues.apache.org/jira/browse/SPARK-3962 Project: Spark I

[jira] [Updated] (SPARK-3962) Mark spark dependency as "provided" in external libraries

2014-10-15 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3962: --- Description: Right now there is not an easy way for users to link against the external stream

[jira] [Resolved] (SPARK-1561) sbt/sbt assembly generates too many local files

2014-10-15 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-1561. Resolution: Not a Problem Does not seem to be an issue any more - resolving per comment from

[jira] [Commented] (SPARK-3431) Parallelize execution of tests

2014-10-15 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173300#comment-14173300 ] Patrick Wendell commented on SPARK-3431: [~srowen] - just wondering, is it trivial

[jira] [Created] (SPARK-3963) Support getting task-scoped properties from TaskContext

2014-10-15 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-3963: -- Summary: Support getting task-scoped properties from TaskContext Key: SPARK-3963 URL: https://issues.apache.org/jira/browse/SPARK-3963 Project: Spark Iss

[jira] [Updated] (SPARK-3963) Support getting task-scoped properties from TaskContext

2014-10-15 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3963: --- Description: This is a proposal for a minor feature. Given stabilization of the TaskContext A

[jira] [Updated] (SPARK-3963) Support getting task-scoped properties from TaskContext

2014-10-15 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3963: --- Description: This is a proposal for a minor feature. Given stabilization of the TaskContext A

[jira] [Commented] (SPARK-3963) Support getting task-scoped properties from TaskContext

2014-10-15 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173404#comment-14173404 ] Patrick Wendell commented on SPARK-3963: [~rxin] and [~adav] I'd be interested in

[jira] [Updated] (SPARK-3963) Support getting task-scoped properties from TaskContext

2014-10-15 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3963: --- Description: This is a proposal for a minor feature. Given stabilization of the TaskContext A

[jira] [Resolved] (SPARK-3874) Provide stable TaskContext API

2014-10-16 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3874. Resolution: Fixed Fix Version/s: 1.2.0 > Provide stable TaskContext API > ---

[jira] [Updated] (SPARK-3975) Block Matrix addition and multiplication

2014-10-16 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3975: --- Component/s: MLlib > Block Matrix addition and multiplication > --

[jira] [Commented] (SPARK-3882) JobProgressListener gets permanently out of sync with long running job

2014-10-16 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174629#comment-14174629 ] Patrick Wendell commented on SPARK-3882: This is a known issue (SPARK-2316) that w

[jira] [Updated] (SPARK-3973) Print callSite information for broadcast variables

2014-10-16 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3973: --- Component/s: Spark Core > Print callSite information for broadcast variables > ---

[jira] [Commented] (SPARK-3963) Support getting task-scoped properties from TaskContext

2014-10-16 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174660#comment-14174660 ] Patrick Wendell commented on SPARK-3963: In the initial version of this - I don't

[jira] [Comment Edited] (SPARK-3963) Support getting task-scoped properties from TaskContext

2014-10-16 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174660#comment-14174660 ] Patrick Wendell edited comment on SPARK-3963 at 10/17/14 2:30 AM: --

[jira] [Comment Edited] (SPARK-3963) Support getting task-scoped properties from TaskContext

2014-10-16 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174660#comment-14174660 ] Patrick Wendell edited comment on SPARK-3963 at 10/17/14 2:35 AM: --

[jira] [Resolved] (SPARK-3855) Binding Exception when running PythonUDFs

2014-10-17 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3855. Resolution: Fixed Fix Version/s: 1.2.0 > Binding Exception when running PythonUDFs >

[jira] [Updated] (SPARK-3135) Avoid memory copy in TorrentBroadcast serialization

2014-10-18 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3135: --- Component/s: Spark Core > Avoid memory copy in TorrentBroadcast serialization > --

[jira] [Updated] (SPARK-3994) countByKey / countByValue do not go through Aggregator

2014-10-18 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3994: --- Component/s: Spark Core > countByKey / countByValue do not go through Aggregator > ---

[jira] [Updated] (SPARK-3694) Allow printing object graph of tasks/RDD's with a debug flag

2014-10-18 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3694: --- Assignee: Ilya Ganelin > Allow printing object graph of tasks/RDD's with a debug flag > --

[jira] [Commented] (SPARK-3694) Allow printing object graph of tasks/RDD's with a debug flag

2014-10-18 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14176110#comment-14176110 ] Patrick Wendell commented on SPARK-3694: All yours! I think this should be accompl

[jira] [Updated] (SPARK-3940) SQL console prints error messages three times

2014-10-18 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3940: --- Summary: SQL console prints error messages three times (was: Print the error code three time

[jira] [Commented] (SPARK-3996) Shade Jetty in Spark deliverables

2014-10-19 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14176485#comment-14176485 ] Patrick Wendell commented on SPARK-3996: This is a good idea if we can do it using

[jira] [Created] (SPARK-4021) Kinesis code can cause compile failures with newer JVM's

2014-10-20 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-4021: -- Summary: Kinesis code can cause compile failures with newer JVM's Key: SPARK-4021 URL: https://issues.apache.org/jira/browse/SPARK-4021 Project: Spark Is

[jira] [Updated] (SPARK-4021) Kinesis code can cause compile failures with newer JDK's

2014-10-20 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4021: --- Summary: Kinesis code can cause compile failures with newer JDK's (was: Kinesis code can caus

[jira] [Commented] (SPARK-4021) Kinesis code can cause compile failures with newer JDK's

2014-10-20 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14177729#comment-14177729 ] Patrick Wendell commented on SPARK-4021: Hey [~srowen] - I am getting this report

[jira] [Updated] (SPARK-4021) Issues observed after upgrading Jenkins to JDK7u71

2014-10-20 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4021: --- Summary: Issues observed after upgrading Jenkins to JDK7u71 (was: Kinesis code can cause comp

[jira] [Comment Edited] (SPARK-4021) Kinesis code can cause compile failures with newer JDK's

2014-10-20 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14177729#comment-14177729 ] Patrick Wendell edited comment on SPARK-4021 at 10/21/14 12:47 AM: -

[jira] [Updated] (SPARK-4021) Issues observed after upgrading Jenkins to JDK7u71

2014-10-20 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4021: --- Component/s: (was: Streaming) Project Infra > Issues observed after upgra

[jira] [Updated] (SPARK-4021) Issues observed after upgrading Jenkins to JDK7u71

2014-10-20 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4021: --- Description: The following compile failure was observed after adding JDK7u71 to Jenkins. Howe

[jira] [Commented] (SPARK-3996) Shade Jetty in Spark deliverables

2014-10-20 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14177741#comment-14177741 ] Patrick Wendell commented on SPARK-3996: I think for this one we should only need

[jira] [Updated] (SPARK-4022) Replace colt dependency (LGPL) with commons-math

2014-10-20 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4022: --- Summary: Replace colt dependency (LGPL) with commons-math (was: Replace colt with commons-mat

[jira] [Created] (SPARK-4022) Replace colt with commons-math

2014-10-20 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-4022: -- Summary: Replace colt with commons-math Key: SPARK-4022 URL: https://issues.apache.org/jira/browse/SPARK-4022 Project: Spark Issue Type: Bug Co

[jira] [Commented] (SPARK-3889) JVM dies with SIGBUS, resulting in ConnectionManager failed ACK

2014-10-20 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14177925#comment-14177925 ] Patrick Wendell commented on SPARK-3889: [~fryz] I think the relevant code path he

[jira] [Updated] (SPARK-3889) JVM dies with SIGBUS, resulting in ConnectionManager failed ACK

2014-10-20 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3889: --- Affects Version/s: (was: 1.1.0) 1.2.0 > JVM dies with SIGBUS, resul

[jira] [Updated] (SPARK-3985) json file path is not right

2014-10-20 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3985: --- Component/s: SQL > json file path is not right > --- > >

[jira] [Updated] (SPARK-4002) JavaKafkaStreamSuite.testKafkaStream fails on OSX

2014-10-20 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4002: --- Component/s: Streaming > JavaKafkaStreamSuite.testKafkaStream fails on OSX > -

[jira] [Updated] (SPARK-4021) Issues observed after upgrading Jenkins to JDK7u71

2014-10-20 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4021: --- Assignee: shane knapp > Issues observed after upgrading Jenkins to JDK7u71 > -

[jira] [Resolved] (SPARK-1042) spark cleans all java broadcast variables when it hits the spark.cleaner.ttl

2014-10-20 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-1042. Resolution: Fixed Fix Version/s: 0.9.2 I think this was fixed back in 0.9.2 > spark

[jira] [Updated] (SPARK-4003) Add {Big Decimal, Timestamp, Date} types to Java SqlContext

2014-10-20 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4003: --- Summary: Add {Big Decimal, Timestamp, Date} types to Java SqlContext (was: Add 3 types for ja

[jira] [Updated] (SPARK-4006) Spark Driver crashes whenever an Executor is registered twice

2014-10-20 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4006: --- Priority: Critical (was: Blocker) Target Version/s: 1.2.0 > Spark Driver crashes

[jira] [Updated] (SPARK-4006) Spark Driver crashes whenever an Executor is registered twice

2014-10-20 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4006: --- Priority: Blocker (was: Critical) > Spark Driver crashes whenever an Executor is registered t

[jira] [Updated] (SPARK-3948) Sort-based shuffle can lead to assorted stream-corruption exceptions

2014-10-20 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3948: --- Priority: Blocker (was: Major) > Sort-based shuffle can lead to assorted stream-corruption ex

[jira] [Commented] (SPARK-4014) TaskContext.attemptId returns taskId

2014-10-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14178053#comment-14178053 ] Patrick Wendell commented on SPARK-4014: [~joshrosen] what do you think about rena

[jira] [Commented] (SPARK-4019) Repartitioning with more than 2000 partitions may drop all data when partitions are mostly empty.

2014-10-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14178056#comment-14178056 ] Patrick Wendell commented on SPARK-4019: Great work getting to the root cause of t

[jira] [Commented] (SPARK-4030) `destroy` method in Broadcast should be public

2014-10-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14178068#comment-14178068 ] Patrick Wendell commented on SPARK-4030: Hey Shivaram - IIRC we made this private

[jira] [Updated] (SPARK-4030) `destroy` method in Broadcast should be public

2014-10-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4030: --- Issue Type: Improvement (was: Bug) > `destroy` method in Broadcast should be public > ---

[jira] [Comment Edited] (SPARK-4030) `destroy` method in Broadcast should be public

2014-10-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14178068#comment-14178068 ] Patrick Wendell edited comment on SPARK-4030 at 10/21/14 7:17 AM: --

[jira] [Comment Edited] (SPARK-4030) `destroy` method in Broadcast should be public

2014-10-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14178068#comment-14178068 ] Patrick Wendell edited comment on SPARK-4030 at 10/21/14 7:17 AM: --

[jira] [Updated] (SPARK-3945) Write properties of hive-site.xml to HiveContext when initilize session state In SparkSQLEnv.scala

2014-10-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3945: --- Assignee: luogankun > Write properties of hive-site.xml to HiveContext when initilize session

[jira] [Commented] (SPARK-3945) Write properties of hive-site.xml to HiveContext when initilize session state In SparkSQLEnv.scala

2014-10-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14178070#comment-14178070 ] Patrick Wendell commented on SPARK-3945: Hey @luogankun would you mind adding a fi

[jira] [Updated] (SPARK-3940) SQL console prints error messages three times

2014-10-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3940: --- Assignee: wangxj > SQL console prints error messages three times > ---

[jira] [Commented] (SPARK-3940) SQL console prints error messages three times

2014-10-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14178073#comment-14178073 ] Patrick Wendell commented on SPARK-3940: Hey [~wangxj8] can you add a first and la

[jira] [Updated] (SPARK-4032) Deprecate YARN support in Spark 1.2

2014-10-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4032: --- Description: When someone builds for yarn alpha, we should just display a warning like {code}

[jira] [Created] (SPARK-4032) Deprecate YARN support in Spark 1.2

2014-10-21 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-4032: -- Summary: Deprecate YARN support in Spark 1.2 Key: SPARK-4032 URL: https://issues.apache.org/jira/browse/SPARK-4032 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-4032) Deprecate YARN support in Spark 1.2

2014-10-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4032: --- Priority: Blocker (was: Major) > Deprecate YARN support in Spark 1.2 > --

[jira] [Resolved] (SPARK-547) Provide a means to package Spark's executor into a tgz

2014-10-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-547. --- Resolution: Fixed This was fixed a long time ago. > Provide a means to package Spark's executo

[jira] [Resolved] (SPARK-773) Add fair scheduler pool information UI similar with hadoop

2014-10-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-773. --- Resolution: Fixed This has existing in Spark for a while - it is a stale issue. > Add fair sch

[jira] [Resolved] (SPARK-890) Allow multiple parallel commands in spark-shell

2014-10-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-890. --- Resolution: Fixed Spark context was made thread-safe a long time ago. > Allow multiple parall

[jira] [Resolved] (SPARK-916) Better Support for Flat/Tabular RDD's

2014-10-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-916. --- Resolution: Fixed Turned out [~marmbrus] did all of this and more in SparkSQL (which btw also

[jira] [Resolved] (SPARK-735) memory leak in KryoSerializer

2014-10-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-735. --- Resolution: Fixed I think this was fixed a long time ago. > memory leak in KryoSerializer > --

[jira] [Reopened] (SPARK-566) Replace polling+sleeping with semaphores in broadcast and shuffle

2014-10-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell reopened SPARK-566: --- > Replace polling+sleeping with semaphores in broadcast and shuffle > -

[jira] [Resolved] (SPARK-566) Replace polling+sleeping with semaphores in broadcast and shuffle

2014-10-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-566. --- Resolution: Not a Problem > Replace polling+sleeping with semaphores in broadcast and shuffle >

[jira] [Resolved] (SPARK-566) Replace polling+sleeping with semaphores in broadcast and shuffle

2014-10-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-566. --- Resolution: Fixed Thew shuffle and broadcast implementations have been re-written at least twic

[jira] [Resolved] (SPARK-909) add task serialization footprint (time and size) into TaskMetrics

2014-10-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-909. --- Resolution: Fixed This has already been fixed. > add task serialization footprint (time and si

[jira] [Updated] (SPARK-3466) Limit size of results that a driver collects for each action

2014-10-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3466: --- Assignee: Davies Liu (was: Matthew Cheah) > Limit size of results that a driver collects for

[jira] [Commented] (SPARK-3466) Limit size of results that a driver collects for each action

2014-10-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14178857#comment-14178857 ] Patrick Wendell commented on SPARK-3466: I spoke with Matt today and I'm re-assign

[jira] [Updated] (SPARK-4033) Integer overflow when SparkPi is called with more than 25000 slices

2014-10-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4033: --- Summary: Integer overflow when SparkPi is called with more than 25000 slices (was: Input of t

[jira] [Updated] (SPARK-4040) calling count() on RDD's emitted from a DStream blocks forEachRDD progress.

2014-10-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4040: --- Component/s: Streaming > calling count() on RDD's emitted from a DStream blocks forEachRDD pro

[jira] [Updated] (SPARK-4043) Add a flag for stopping threads of cancelled tasks if Thread.interrupt doesn't kill them

2014-10-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4043: --- Component/s: Spark Core > Add a flag for stopping threads of cancelled tasks if Thread.interru

[jira] [Resolved] (SPARK-1813) Add a utility to SparkConf that makes using Kryo really easy

2014-10-21 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-1813. Resolution: Fixed Fix Version/s: 1.2.0 Assignee: Sandy Ryza Fixed in https:/

[jira] [Resolved] (SPARK-3812) Adapt maven build to publish effective pom.

2014-10-22 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3812. Resolution: Fixed Assignee: Prashant Sharma Fixed by: https://github.com/apache/spark/

[jira] [Commented] (SPARK-3655) Secondary sort

2014-10-22 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14181066#comment-14181066 ] Patrick Wendell commented on SPARK-3655: Hey [~koertkuipers] - i'm not an expert o

[jira] [Updated] (SPARK-4020) Failed executor not properly removed if it has not run tasks

2014-10-22 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4020: --- Component/s: Spark Core > Failed executor not properly removed if it has not run tasks > -

[jira] [Commented] (SPARK-3655) Secondary sort

2014-10-23 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14181792#comment-14181792 ] Patrick Wendell commented on SPARK-3655: Yeah so to be clear here is what I meant:

[jira] [Commented] (SPARK-1239) Don't fetch all map output statuses at each reducer during shuffles

2014-10-23 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182122#comment-14182122 ] Patrick Wendell commented on SPARK-1239: Hey Kostas - there are a few other bugs t

[jira] [Reopened] (SPARK-3812) Adapt maven build to publish effective pom.

2014-10-23 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell reopened SPARK-3812: It appeared that this was creating an issue with the maven tests. I am reverting this to see wh

[jira] [Commented] (SPARK-4030) `destroy` method in Broadcast should be public

2014-10-23 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182175#comment-14182175 ] Patrick Wendell commented on SPARK-4030: I'm fine to open it up. I do think destro

[jira] [Resolved] (SPARK-4019) Shuffling with more than 2000 reducers may drop all data when partitons are mostly empty or cause deserialization errors if at least one partition is empty

2014-10-23 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-4019. Resolution: Fixed Fix Version/s: 1.2.0 Fixed by Josh's patch: https://github.com/apac

[jira] [Created] (SPARK-4073) Parquet+Snappy can cause significant off-heap memory usage

2014-10-23 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-4073: -- Summary: Parquet+Snappy can cause significant off-heap memory usage Key: SPARK-4073 URL: https://issues.apache.org/jira/browse/SPARK-4073 Project: Spark

[jira] [Resolved] (SPARK-3812) Adapt maven build to publish effective pom.

2014-10-23 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-3812. Resolution: Fixed Fix Version/s: 1.2.0 Okay, let's try again. > Adapt maven build to

[jira] [Commented] (SPARK-3561) Allow for pluggable execution contexts in Spark

2014-10-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182511#comment-14182511 ] Patrick Wendell commented on SPARK-3561: Hey [~ozhurakousky] - adding an @Experime

[jira] [Commented] (SPARK-3561) Allow for pluggable execution contexts in Spark

2014-10-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14182520#comment-14182520 ] Patrick Wendell commented on SPARK-3561: One other thing - if projects really do w

[jira] [Commented] (SPARK-4066) Make whether maven builds fails on scalastyle violation configurable

2014-10-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183235#comment-14183235 ] Patrick Wendell commented on SPARK-4066: [~srowen] I don't see a good argument for

[jira] [Commented] (SPARK-4079) Snappy bundled with Spark does not work on older Linux distributions

2014-10-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183317#comment-14183317 ] Patrick Wendell commented on SPARK-4079: What about just catching the exception an

[jira] [Commented] (SPARK-4066) Make whether maven builds fails on scalastyle violation configurable

2014-10-24 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183399#comment-14183399 ] Patrick Wendell commented on SPARK-4066: That was actually my thought originally -

[jira] [Updated] (SPARK-4064) NioBlockTransferService should deal with empty messages correctly

2014-10-26 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4064: --- Summary: NioBlockTransferService should deal with empty messages correctly (was: If we create

[jira] [Commented] (SPARK-3655) Secondary sort

2014-10-26 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184565#comment-14184565 ] Patrick Wendell commented on SPARK-3655: Okay, sounds good. > Secondary sort > --

[jira] [Updated] (SPARK-3655) Support sorting of values in addition to keys (i.e. secondary sort)

2014-10-26 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3655: --- Summary: Support sorting of values in addition to keys (i.e. secondary sort) (was: Secondary

[jira] [Updated] (SPARK-4056) Upgrade snappy-java to 1.1.1.5

2014-10-26 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4056: --- Component/s: Spark Core > Upgrade snappy-java to 1.1.1.5 > -- > >

[jira] [Updated] (SPARK-4085) Job will fail if a shuffle file that's read locally gets deleted

2014-10-26 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-4085: --- Component/s: Spark Core > Job will fail if a shuffle file that's read locally gets deleted > -

[jira] [Updated] (SPARK-2760) Caching tables from multiple databases does not work

2014-10-26 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2760: --- Component/s: SQL > Caching tables from multiple databases does not work >

[jira] [Updated] (SPARK-3917) Compress data before network transfer

2014-10-26 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3917: --- Priority: Major (was: Critical) > Compress data before network transfer > ---

[jira] [Commented] (SPARK-2532) Fix issues with consolidated shuffle

2014-10-26 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184576#comment-14184576 ] Patrick Wendell commented on SPARK-2532: Hey [~matei] - you created some sub-tasks

[jira] [Resolved] (SPARK-2633) enhance spark listener API to gather more spark job information

2014-10-26 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-2633. Resolution: Duplicate I believe the design of SPARK-2321 is such that it covers Hive's use c

[jira] [Comment Edited] (SPARK-3962) Mark spark dependency as "provided" in external libraries

2014-10-26 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184580#comment-14184580 ] Patrick Wendell edited comment on SPARK-3962 at 10/26/14 6:11 PM: --

[jira] [Updated] (SPARK-3962) Mark spark dependency as "provided" in external libraries

2014-10-26 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-3962: --- Assignee: Prashant Sharma > Mark spark dependency as "provided" in external libraries > --

[jira] [Commented] (SPARK-3962) Mark spark dependency as "provided" in external libraries

2014-10-26 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184580#comment-14184580 ] Patrick Wendell commented on SPARK-3962: [~prashant_] can you take a crack at this

[jira] [Created] (SPARK-4092) Input metrics don't work for coalesce()'d RDD's

2014-10-26 Thread Patrick Wendell (JIRA)
Patrick Wendell created SPARK-4092: -- Summary: Input metrics don't work for coalesce()'d RDD's Key: SPARK-4092 URL: https://issues.apache.org/jira/browse/SPARK-4092 Project: Spark Issue Type:

[jira] [Commented] (SPARK-3266) JavaDoubleRDD doesn't contain max()

2014-10-26 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184704#comment-14184704 ] Patrick Wendell commented on SPARK-3266: I think it sort of depends how many peopl

<    1   2   3   4   5   6   7   8   9   10   >