[jira] [Comment Edited] (SPARK-15823) Add @property for 'accuracy' in MulticlassMetrics

2016-06-09 Thread zhengruifeng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322308#comment-15322308 ] zhengruifeng edited comment on SPARK-15823 at 6/9/16 10:20 AM: ---

[jira] [Commented] (SPARK-15823) Add @property for 'accuracy' in MulticlassMetrics

2016-06-09 Thread zhengruifeng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322309#comment-15322309 ] zhengruifeng commented on SPARK-15823: -- {MulticlassMetrics.confusionMatrix} may need {@property}

[jira] [Issue Comment Deleted] (SPARK-15823) Add @property for 'accuracy' in MulticlassMetrics

2016-06-09 Thread zhengruifeng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng updated SPARK-15823: - Comment: was deleted (was: {MulticlassMetrics.confusionMatrix} may need {@property} too, but I

[jira] [Created] (SPARK-15842) Add support for socket stream.

2016-06-09 Thread Prashant Sharma (JIRA)
Prashant Sharma created SPARK-15842: --- Summary: Add support for socket stream. Key: SPARK-15842 URL: https://issues.apache.org/jira/browse/SPARK-15842 Project: Spark Issue Type: Sub-task

[jira] [Comment Edited] (SPARK-15545) R remove non-exported unused methods, like jsonRDD

2016-06-09 Thread Miao Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322039#comment-15322039 ] Miao Wang edited comment on SPARK-15545 at 6/9/16 6:56 AM: --- [~shivaram]Thanks

[jira] [Commented] (SPARK-15086) Update Java API once the Scala one is finalized

2016-06-09 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322082#comment-15322082 ] Weichen Xu commented on SPARK-15086: OK. [~srowen] What do you think about it? > Update Java API

[jira] [Commented] (SPARK-15823) Add @property for 'accuracy' in MulticlassMetrics

2016-06-09 Thread zhengruifeng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322308#comment-15322308 ] zhengruifeng commented on SPARK-15823: -- {MulticlassMetrics.confusionMatrix} may need {@property}

[jira] [Commented] (SPARK-15781) Reduce spark.memory.fraction default to avoid overrunning old gen in JVM default config

2016-06-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322425#comment-15322425 ] Sean Owen commented on SPARK-15781: --- Oh I'm sorry, I put this comment on entirely the wrong JIRA -- too

[jira] [Updated] (SPARK-15781) Misleading deprecated property in standalone cluster configuration documentation

2016-06-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-15781: -- Summary: Misleading deprecated property in standalone cluster configuration documentation (was:

[jira] [Issue Comment Deleted] (SPARK-15781) Misleading deprecated property in standalone cluster configuration documentation

2016-06-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-15781: -- Comment: was deleted (was: PS [~JonathanTaws] do you have some output from -verbose:gc that might

[jira] [Commented] (SPARK-11765) Avoid assign UI port between browser unsafe ports (or just 4045: lockd)

2016-06-09 Thread Willy Lee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322442#comment-15322442 ] Willy Lee commented on SPARK-11765: --- As of what version? I'll want to have our team upgrade. > Avoid

[jira] [Assigned] (SPARK-15840) New csv reader does not "determine the input schema"

2016-06-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15840: Assignee: (was: Apache Spark) > New csv reader does not "determine the input schema"

[jira] [Assigned] (SPARK-15840) New csv reader does not "determine the input schema"

2016-06-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15840: Assignee: Apache Spark > New csv reader does not "determine the input schema" >

[jira] [Commented] (SPARK-15840) New csv reader does not "determine the input schema"

2016-06-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322440#comment-15322440 ] Apache Spark commented on SPARK-15840: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Updated] (SPARK-15823) Add @property for 'accuracy' in MulticlassMetrics

2016-06-09 Thread zhengruifeng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng updated SPARK-15823: - Summary: Add @property for 'accuracy' in MulticlassMetrics (was: Add @property for 'property'

[jira] [Commented] (SPARK-15781) Reduce spark.memory.fraction default to avoid overrunning old gen in JVM default config

2016-06-09 Thread Jonathan Taws (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322324#comment-15322324 ] Jonathan Taws commented on SPARK-15781: --- By launching a session with {{SPARK_WORKER_INSTANCES}} set

[jira] [Updated] (SPARK-15796) Reduce spark.memory.fraction default to avoid overrunning old gen in JVM default config

2016-06-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-15796: -- Summary: Reduce spark.memory.fraction default to avoid overrunning old gen in JVM default config

[jira] [Commented] (SPARK-15796) Reduce spark.memory.fraction default to avoid overrunning old gen in JVM default config

2016-06-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322426#comment-15322426 ] Sean Owen commented on SPARK-15796: --- PS [~gfeher] do you have some output from -verbose:gc that might

[jira] [Commented] (SPARK-1882) Support dynamic memory sharing in Mesos

2016-06-09 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322364#comment-15322364 ] Stavros Kontopoulos commented on SPARK-1882: Does dynamic allocation help with the

[jira] [Updated] (SPARK-15842) Add support for socket stream.

2016-06-09 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-15842: Description: Streaming so far has an offset based sources with all the available sources

[jira] [Assigned] (SPARK-15842) Add support for socket stream.

2016-06-09 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma reassigned SPARK-15842: --- Assignee: Prashant Sharma > Add support for socket stream. >

[jira] [Updated] (SPARK-15831) Kryo 2.21 TreeMap serialization bug causes random job failures with RDDs of HBase puts

2016-06-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-15831: -- Affects Version/s: 1.5.2 1.6.1 Target Version/s: (was: 1.5.0) > Kryo

[jira] [Commented] (SPARK-15801) spark-submit --num-executors switch also works without YARN

2016-06-09 Thread Jonathan Taws (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322322#comment-15322322 ] Jonathan Taws commented on SPARK-15801: --- I don't think it is a problem, but it might be interesting

[jira] [Updated] (SPARK-15472) Add support for writing in `csv`, `json`, `text` formats in Structured Streaming

2016-06-09 Thread Liwei Lin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liwei Lin updated SPARK-15472: -- Summary: Add support for writing in `csv`, `json`, `text` formats in Structured Streaming (was: Add

[jira] [Commented] (SPARK-15472) Add partitioned `csv`, `json`, `text` format support for FileStreamSink

2016-06-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322325#comment-15322325 ] Apache Spark commented on SPARK-15472: -- User 'lw-lin' has created a pull request for this issue:

[jira] [Resolved] (SPARK-12712) test-dependencies.sh script fails when run against empty .m2 cache

2016-06-09 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-12712. Resolution: Fixed Fix Version/s: 1.6.2 2.0.0 Issue resolved by pull

[jira] [Commented] (SPARK-15840) New csv reader does not "determine the input schema"

2016-06-09 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-15840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322142#comment-15322142 ] Ernst Sjöstrand commented on SPARK-15840: - Also, the documentation implies that an inferSchema

[jira] [Updated] (SPARK-15697) [SPARK REPL] unblock some of the useful repl commands.

2016-06-09 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-15697: Description: "implicits", "javap", "power", "type", "kind" commands in repl are blocked.

[jira] [Updated] (SPARK-15697) [SPARK REPL] unblock some of the useful repl commands.

2016-06-09 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-15697: Description: "implicits", "javap", "power", "type", "kind" commands in repl are blocked.

[jira] [Updated] (SPARK-15841) [SPARK REPL] REPLSuite has in correct env set for a couple of tests.

2016-06-09 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-15841: Component/s: Spark Shell > [SPARK REPL] REPLSuite has in correct env set for a couple of

[jira] [Created] (SPARK-15841) [SPARK REPL] REPLSuite has in correct env set for a couple of tests.

2016-06-09 Thread Prashant Sharma (JIRA)
Prashant Sharma created SPARK-15841: --- Summary: [SPARK REPL] REPLSuite has in correct env set for a couple of tests. Key: SPARK-15841 URL: https://issues.apache.org/jira/browse/SPARK-15841 Project:

[jira] [Resolved] (SPARK-15802) SparkSQL connection fail using shell command "bin/beeline -u "jdbc:hive2://*.*.*.*:10000/default""

2016-06-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-15802. --- Resolution: Not A Problem It looks like you show the answer in your question, not sure what you're

[jira] [Commented] (SPARK-15086) Update Java API once the Scala one is finalized

2016-06-09 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322075#comment-15322075 ] Reynold Xin commented on SPARK-15086: - I was suggesting renaming both, so the two would be

[jira] [Commented] (SPARK-15086) Update Java API once the Scala one is finalized

2016-06-09 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322074#comment-15322074 ] Weichen Xu commented on SPARK-15086: If do so, only rename the java API in this type or rename scala

[jira] [Commented] (SPARK-15839) Maven doc JAR generation fails when JAVA_7_HOME is set

2016-06-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322103#comment-15322103 ] Apache Spark commented on SPARK-15839: -- User 'JoshRosen' has created a pull request for this issue:

[jira] [Commented] (SPARK-15840) New csv reader does not "determine the input schema"

2016-06-09 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-15840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322127#comment-15322127 ] Ernst Sjöstrand commented on SPARK-15840: - The old databricks csv had an option called

[jira] [Commented] (SPARK-15840) New csv reader does not "determine the input schema"

2016-06-09 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-15840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322143#comment-15322143 ] Ernst Sjöstrand commented on SPARK-15840: - I have only tested this for Python, not sure if it

[jira] [Comment Edited] (SPARK-15840) New csv reader does not "determine the input schema"

2016-06-09 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322140#comment-15322140 ] Hyukjin Kwon edited comment on SPARK-15840 at 6/9/16 8:24 AM: -- There is

[jira] [Resolved] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-15716. --- Resolution: Not A Problem > Memory usage of driver keeps growing up in Spark Streaming >

[jira] [Created] (SPARK-15839) Maven doc JAR generation fails when JAVA_7_HOME is set

2016-06-09 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-15839: -- Summary: Maven doc JAR generation fails when JAVA_7_HOME is set Key: SPARK-15839 URL: https://issues.apache.org/jira/browse/SPARK-15839 Project: Spark Issue

[jira] [Created] (SPARK-15840) New csv reader does not "determine the input schema"

2016-06-09 Thread JIRA
Ernst Sjöstrand created SPARK-15840: --- Summary: New csv reader does not "determine the input schema" Key: SPARK-15840 URL: https://issues.apache.org/jira/browse/SPARK-15840 Project: Spark

[jira] [Updated] (SPARK-15840) New csv reader does not "determine the input schema"

2016-06-09 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-15840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ernst Sjöstrand updated SPARK-15840: Description: When testing the new csv reader I found that it would not determine the input

[jira] [Updated] (SPARK-15841) [SPARK REPL] REPLSuite has incorrect env set for a couple of tests.

2016-06-09 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-15841: Summary: [SPARK REPL] REPLSuite has incorrect env set for a couple of tests. (was: [SPARK

[jira] [Commented] (SPARK-15837) PySpark ML Word2Vec should support maxSentenceLength

2016-06-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322195#comment-15322195 ] Sean Owen commented on SPARK-15837: --- Yeah, ideally we would have suggested and done this in the first

[jira] [Commented] (SPARK-15841) [SPARK REPL] REPLSuite has incorrect env set for a couple of tests.

2016-06-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322199#comment-15322199 ] Apache Spark commented on SPARK-15841: -- User 'ScrapCodes' has created a pull request for this issue:

[jira] [Assigned] (SPARK-15841) [SPARK REPL] REPLSuite has incorrect env set for a couple of tests.

2016-06-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15841: Assignee: (was: Apache Spark) > [SPARK REPL] REPLSuite has incorrect env set for a

[jira] [Commented] (SPARK-11765) Avoid assign UI port between browser unsafe ports (or just 4045: lockd)

2016-06-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322180#comment-15322180 ] Sean Owen commented on SPARK-11765: --- That's how it works now. > Avoid assign UI port between browser

[jira] [Resolved] (SPARK-15801) spark-submit --num-executors switch also works without YARN

2016-06-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-15801. --- Resolution: Not A Problem This much seems to be not a problem. > spark-submit --num-executors

[jira] [Commented] (SPARK-15840) New csv reader does not "determine the input schema"

2016-06-09 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-15840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322137#comment-15322137 ] Ernst Sjöstrand commented on SPARK-15840: - Perhaps related to SPARK-13667 ? > New csv reader

[jira] [Updated] (SPARK-15840) New csv reader does not "determine the input schema"

2016-06-09 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-15840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ernst Sjöstrand updated SPARK-15840: Description: When testing the new csv reader I found that it would not determine the input

[jira] [Resolved] (SPARK-15836) Spark 2.0/master maven snapshots are broken

2016-06-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-15836. --- Resolution: Duplicate Target Version/s: (was: 2.0.0) > Spark 2.0/master maven snapshots

[jira] [Assigned] (SPARK-15841) [SPARK REPL] REPLSuite has incorrect env set for a couple of tests.

2016-06-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15841: Assignee: Apache Spark > [SPARK REPL] REPLSuite has incorrect env set for a couple of

[jira] [Resolved] (SPARK-15818) Upgrade to Hadoop 2.7.2

2016-06-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-15818. --- Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 13556

[jira] [Updated] (SPARK-15818) Upgrade to Hadoop 2.7.2

2016-06-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-15818: -- Assignee: Adam Roberts > Upgrade to Hadoop 2.7.2 > --- > > Key:

[jira] [Commented] (SPARK-15840) New csv reader does not "determine the input schema"

2016-06-09 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322140#comment-15322140 ] Hyukjin Kwon commented on SPARK-15840: -- There is {{inferSchema}} option but it seems it was missed

[jira] [Commented] (SPARK-15840) New csv reader does not "determine the input schema"

2016-06-09 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322147#comment-15322147 ] Hyukjin Kwon commented on SPARK-15840: -- For custom dateFormat, here there are,

[jira] [Updated] (SPARK-15697) [SPARK REPL] unblock some of the useful repl commands.

2016-06-09 Thread Prashant Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-15697: Description: "implicits", "javap", "power", "type", "kind" commands in repl are blocked.

[jira] [Updated] (SPARK-15781) Reduce spark.memory.fraction default to avoid overrunning old gen in JVM default config

2016-06-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-15781: -- Summary: Reduce spark.memory.fraction default to avoid overrunning old gen in JVM default config

[jira] [Commented] (SPARK-11765) Avoid assign UI port between browser unsafe ports (or just 4045: lockd)

2016-06-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322494#comment-15322494 ] Sean Owen commented on SPARK-11765: --- You can do two things -- pick another starting port, or limit the

[jira] [Commented] (SPARK-11765) Avoid assign UI port between browser unsafe ports (or just 4045: lockd)

2016-06-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322446#comment-15322446 ] Sean Owen commented on SPARK-11765: --- For as long as I can remember it has iterated through several next

[jira] [Commented] (SPARK-15585) Don't use null in data source options to indicate default value

2016-06-09 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322670#comment-15322670 ] Takeshi Yamamuro commented on SPARK-15585: -- I'm afraid the `sep` option for `csv` overrides the

[jira] [Commented] (SPARK-15772) Improve Scala API docs

2016-06-09 Thread nirav patel (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322709#comment-15322709 ] nirav patel commented on SPARK-15772: - I can't point you to every individual functions which needs

[jira] [Commented] (SPARK-15837) PySpark ML Word2Vec should support maxSentenceLength

2016-06-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322720#comment-15322720 ] Apache Spark commented on SPARK-15837: -- User 'WeichenXu123' has created a pull request for this

[jira] [Assigned] (SPARK-15837) PySpark ML Word2Vec should support maxSentenceLength

2016-06-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15837: Assignee: Apache Spark > PySpark ML Word2Vec should support maxSentenceLength >

[jira] [Assigned] (SPARK-15837) PySpark ML Word2Vec should support maxSentenceLength

2016-06-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15837: Assignee: (was: Apache Spark) > PySpark ML Word2Vec should support maxSentenceLength

[jira] [Commented] (SPARK-2984) FileNotFoundException on _temporary directory

2016-06-09 Thread Sandeep (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322739#comment-15322739 ] Sandeep commented on SPARK-2984: I tried with spark.speculation=false as well and it still gives the same

[jira] [Commented] (SPARK-15844) HistoryServer doesn't come up if spark.authenticate = true

2016-06-09 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322774#comment-15322774 ] Steve Loughran commented on SPARK-15844: Stack. {code} 16/05/31 22:46:25 INFO SecurityManager:

[jira] [Created] (SPARK-15844) HistoryServer doesn't come up if spark.authenticate = true

2016-06-09 Thread Steve Loughran (JIRA)
Steve Loughran created SPARK-15844: -- Summary: HistoryServer doesn't come up if spark.authenticate = true Key: SPARK-15844 URL: https://issues.apache.org/jira/browse/SPARK-15844 Project: Spark

[jira] [Commented] (SPARK-15828) YARN is not aware of Spark's External Shuffle Service

2016-06-09 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322892#comment-15322892 ] Saisai Shao commented on SPARK-15828: - OK, I guess you're running on AWS or similar cloud

[jira] [Commented] (SPARK-14485) Task finished cause fetch failure when its executor has already been removed by driver

2016-06-09 Thread Kay Ousterhout (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322893#comment-15322893 ] Kay Ousterhout commented on SPARK-14485: I don't think (a) is especially rare: that's the case

[jira] [Commented] (SPARK-14485) Task finished cause fetch failure when its executor has already been removed by driver

2016-06-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322894#comment-15322894 ] Apache Spark commented on SPARK-14485: -- User 'kayousterhout' has created a pull request for this

[jira] [Commented] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-09 Thread Yan Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322912#comment-15322912 ] Yan Chen commented on SPARK-15716: -- Original problem comes from Hortonworks. We also tried to use

[jira] [Commented] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-09 Thread Yan Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322920#comment-15322920 ] Yan Chen commented on SPARK-15716: -- [~srowen] Could I know why this issue is closed? > Memory usage of

[jira] [Comment Edited] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-09 Thread Yan Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322912#comment-15322912 ] Yan Chen edited comment on SPARK-15716 at 6/9/16 5:34 PM: -- Original problem

[jira] [Comment Edited] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-09 Thread Yan Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322920#comment-15322920 ] Yan Chen edited comment on SPARK-15716 at 6/9/16 5:34 PM: -- [~srowen] Could I

[jira] [Commented] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-09 Thread Yan Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322930#comment-15322930 ] Yan Chen commented on SPARK-15716: -- Why it is "not a problem" even if it crashes the streaming process?

[jira] [Commented] (SPARK-15828) YARN is not aware of Spark's External Shuffle Service

2016-06-09 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322788#comment-15322788 ] Saisai Shao commented on SPARK-15828: - I think this issue is not related to dynamic allocation, if

[jira] [Commented] (SPARK-15801) spark-submit --num-executors switch also works without YARN

2016-06-09 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322800#comment-15322800 ] Saisai Shao commented on SPARK-15801: - It has already been mentioned in {{spark-submit --help}}:

[jira] [Commented] (SPARK-15800) Accessing kerberised hdfs from Spark running with Resource Manager

2016-06-09 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322804#comment-15322804 ] Saisai Shao commented on SPARK-15800: - {quote} Spark is currently running using the Resource Manager,

[jira] [Updated] (SPARK-15845) Expose metrics for sub-stage transformations and action

2016-06-09 Thread nirav patel (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nirav patel updated SPARK-15845: Description: Spark optimizes DAG processing by efficiently selecting stage boundaries. This

[jira] [Resolved] (SPARK-15804) Manually added metadata not saving with parquet

2016-06-09 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-15804. - Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 13555

[jira] [Resolved] (SPARK-15788) PySpark IDFModel missing "idf" property

2016-06-09 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath resolved SPARK-15788. Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 13540

[jira] [Commented] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-09 Thread Yan Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322959#comment-15322959 ] Yan Chen commented on SPARK-15716: -- I've marked it as invalid since it is a problem, but you think it is

[jira] [Comment Edited] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-09 Thread Yan Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322959#comment-15322959 ] Yan Chen edited comment on SPARK-15716 at 6/9/16 5:45 PM: -- I've marked it as

[jira] [Comment Edited] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-09 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322982#comment-15322982 ] Shixiong Zhu edited comment on SPARK-15716 at 6/9/16 5:54 PM: -- If possible,

[jira] [Updated] (SPARK-15433) PySpark core test should not use SerDe from PythonMLLibAPI

2016-06-09 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-15433: --- Assignee: Liang-Chi Hsieh > PySpark core test should not use SerDe from PythonMLLibAPI >

[jira] [Commented] (SPARK-15828) YARN is not aware of Spark's External Shuffle Service

2016-06-09 Thread Miles Crawford (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323043#comment-15323043 ] Miles Crawford commented on SPARK-15828: That's correct, on a cloud provider, AWS to be specific.

[jira] [Commented] (SPARK-15781) Misleading deprecated property in standalone cluster configuration documentation

2016-06-09 Thread Jonathan Taws (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323235#comment-15323235 ] Jonathan Taws commented on SPARK-15781: --- What are our nextsteps on this ? CC Andrew or someone who

[jira] [Commented] (SPARK-15801) spark-submit --num-executors switch also works without YARN

2016-06-09 Thread Jonathan Taws (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323239#comment-15323239 ] Jonathan Taws commented on SPARK-15801: --- Indeed, should be enough as it is then. > spark-submit

[jira] [Commented] (SPARK-15801) spark-submit --num-executors switch also works without YARN

2016-06-09 Thread Jonathan Taws (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323242#comment-15323242 ] Jonathan Taws commented on SPARK-15801: --- Indeed, should be enough as it is then. > spark-submit

[jira] [Comment Edited] (SPARK-15781) Misleading deprecated property in standalone cluster configuration documentation

2016-06-09 Thread Jonathan Taws (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323235#comment-15323235 ] Jonathan Taws edited comment on SPARK-15781 at 6/9/16 8:11 PM: --- What are

[jira] [Issue Comment Deleted] (SPARK-15801) spark-submit --num-executors switch also works without YARN

2016-06-09 Thread Jonathan Taws (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Taws updated SPARK-15801: -- Comment: was deleted (was: Indeed, should be enough as it is then. ) > spark-submit

[jira] [Commented] (SPARK-14485) Task finished cause fetch failure when its executor has already been removed by driver

2016-06-09 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323344#comment-15323344 ] Marcelo Vanzin commented on SPARK-14485: bq. I don't think (a) is especially rare: that's the

[jira] [Commented] (SPARK-14321) Reduce date format cost in date functions

2016-06-09 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323384#comment-15323384 ] Apache Spark commented on SPARK-14321: -- User 'hvanhovell' has created a pull request for this issue:

[jira] [Commented] (SPARK-15613) Incorrect days to millis conversion

2016-06-09 Thread Bo Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323386#comment-15323386 ] Bo Meng commented on SPARK-15613: - Does this only happen to 1.6? I have tried on the latest master and it

[jira] [Comment Edited] (SPARK-15613) Incorrect days to millis conversion

2016-06-09 Thread Bo Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323386#comment-15323386 ] Bo Meng edited comment on SPARK-15613 at 6/9/16 9:24 PM: - Does this only happen

[jira] [Commented] (SPARK-15716) Memory usage of driver keeps growing up in Spark Streaming

2016-06-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323019#comment-15323019 ] Sean Owen commented on SPARK-15716: --- I agree you obviously have some issue in your system. The question

[jira] [Commented] (SPARK-15842) Add support for socket stream.

2016-06-09 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323006#comment-15323006 ] Tathagata Das commented on SPARK-15842: --- This is slightly at odds with the fundamental design of

[jira] [Updated] (SPARK-15847) DecisionTreeRunner example stucks with "NoClassDefFoundError: org/apache/avro/generic/GenericRecord"

2016-06-09 Thread Yesha Vora (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yesha Vora updated SPARK-15847: --- Affects Version/s: 2.0.0 > DecisionTreeRunner example stucks with "NoClassDefFoundError: >

[jira] [Created] (SPARK-15848) Spark unable to read partitioned table in avro format and column name in upper case

2016-06-09 Thread Zhan Zhang (JIRA)
Zhan Zhang created SPARK-15848: -- Summary: Spark unable to read partitioned table in avro format and column name in upper case Key: SPARK-15848 URL: https://issues.apache.org/jira/browse/SPARK-15848

  1   2   3   >