[jira] [Resolved] (SPARK-15706) Wrong Answer when using IF NOT EXISTS in INSERT OVERWRITE for DYNAMIC PARTITION

2016-06-16 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai resolved SPARK-15706. -- Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 13447

[jira] [Updated] (SPARK-15706) Wrong Answer when using IF NOT EXISTS in INSERT OVERWRITE for DYNAMIC PARTITION

2016-06-16 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-15706: - Assignee: Xiao Li > Wrong Answer when using IF NOT EXISTS in INSERT OVERWRITE for DYNAMIC > PARTITION >

[jira] [Updated] (SPARK-15989) PySpark SQL python-only UDTs don't support nested types

2016-06-16 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell updated SPARK-15989: -- Priority: Major (was: Blocker) > PySpark SQL python-only UDTs don't support nested

[jira] [Created] (SPARK-16012) add gapplyCollect() for SparkDataFrame

2016-06-16 Thread Sun Rui (JIRA)
Sun Rui created SPARK-16012: --- Summary: add gapplyCollect() for SparkDataFrame Key: SPARK-16012 URL: https://issues.apache.org/jira/browse/SPARK-16012 Project: Spark Issue Type: Sub-task

[jira] [Assigned] (SPARK-16006) Attemping to write empty DataFrame with no fields throw non-intuitive exception

2016-06-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16006: Assignee: Apache Spark > Attemping to write empty DataFrame with no fields throw

[jira] [Assigned] (SPARK-16006) Attemping to write empty DataFrame with no fields throw non-intuitive exception

2016-06-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16006: Assignee: (was: Apache Spark) > Attemping to write empty DataFrame with no fields

[jira] [Commented] (SPARK-16006) Attemping to write empty DataFrame with no fields throw non-intuitive exception

2016-06-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335450#comment-15335450 ] Apache Spark commented on SPARK-16006: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Created] (SPARK-16011) SQL metrics include duplicated attempts

2016-06-16 Thread Davies Liu (JIRA)
Davies Liu created SPARK-16011: -- Summary: SQL metrics include duplicated attempts Key: SPARK-16011 URL: https://issues.apache.org/jira/browse/SPARK-16011 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

2016-06-16 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-15822. Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 13723

[jira] [Commented] (SPARK-14048) Aggregation operations on structs fail when the structs have fields with special characters

2016-06-16 Thread Simeon Simeonov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335419#comment-15335419 ] Simeon Simeonov commented on SPARK-14048: - I can confirm that this workaround works. >

[jira] [Commented] (SPARK-16007) Empty DataFrame created with spark.read.csv() does not respect user specified schema

2016-06-16 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335404#comment-15335404 ] Xiao Li commented on SPARK-16007: - Sure, I will add test cases after the SPARK-15982 is resolved.

[jira] [Commented] (SPARK-16007) Empty DataFrame created with spark.read.csv() does not respect user specified schema

2016-06-16 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335400#comment-15335400 ] Tathagata Das commented on SPARK-16007: --- This may have been fixed by the changes in SPARK-15982,

[jira] [Closed] (SPARK-15947) Make pipeline components backward compatible with old vector columns in Scala/Java

2016-06-16 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng closed SPARK-15947. - > Make pipeline components backward compatible with old vector columns in > Scala/Java >

[jira] [Assigned] (SPARK-15946) Wrap the conversion utils in Python

2016-06-16 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-15946: - Assignee: Xiangrui Meng > Wrap the conversion utils in Python >

[jira] [Commented] (SPARK-12177) Update KafkaDStreams to new Kafka 0.10 Consumer API

2016-06-16 Thread Jinxia Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335369#comment-15335369 ] Jinxia Liu commented on SPARK-12177: Thanks Cody, I will check your example. I can move the extra

[jira] [Commented] (SPARK-12177) Update KafkaDStreams to new Kafka 0.10 Consumer API

2016-06-16 Thread Jinxia Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335363#comment-15335363 ] Jinxia Liu commented on SPARK-12177: Okay if you think its ok then let it be. > Update KafkaDStreams

[jira] [Assigned] (SPARK-16008) ML Logistic Regression aggregator serializes unnecessary data

2016-06-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16008: Assignee: (was: Apache Spark) > ML Logistic Regression aggregator serializes

[jira] [Assigned] (SPARK-16008) ML Logistic Regression aggregator serializes unnecessary data

2016-06-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16008: Assignee: Apache Spark > ML Logistic Regression aggregator serializes unnecessary data >

[jira] [Updated] (SPARK-16008) ML Logistic Regression aggregator serializes unnecessary data

2016-06-16 Thread Seth Hendrickson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Hendrickson updated SPARK-16008: - Issue Type: Improvement (was: Bug) > ML Logistic Regression aggregator serializes

[jira] [Commented] (SPARK-16008) ML Logistic Regression aggregator serializes unnecessary data

2016-06-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335359#comment-15335359 ] Apache Spark commented on SPARK-16008: -- User 'sethah' has created a pull request for this issue:

[jira] [Assigned] (SPARK-16010) Code Refactoring, Test Case Improvement and Description Updates for SQLConf spark.sql.parquet.filterPushdown

2016-06-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16010: Assignee: (was: Apache Spark) > Code Refactoring, Test Case Improvement and

[jira] [Assigned] (SPARK-16010) Code Refactoring, Test Case Improvement and Description Updates for SQLConf spark.sql.parquet.filterPushdown

2016-06-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16010: Assignee: Apache Spark > Code Refactoring, Test Case Improvement and Description Updates

[jira] [Commented] (SPARK-16010) Code Refactoring, Test Case Improvement and Description Updates for SQLConf spark.sql.parquet.filterPushdown

2016-06-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335354#comment-15335354 ] Apache Spark commented on SPARK-16010: -- User 'gatorsmile' has created a pull request for this issue:

[jira] [Created] (SPARK-16010) Code Refactoring, Test Case Improvement and Description Updates for SQLConf spark.sql.parquet.filterPushdown

2016-06-16 Thread Xiao Li (JIRA)
Xiao Li created SPARK-16010: --- Summary: Code Refactoring, Test Case Improvement and Description Updates for SQLConf spark.sql.parquet.filterPushdown Key: SPARK-16010 URL:

[jira] [Assigned] (SPARK-16009) DataFrameRead.json(path) compatibility broken with Spark 1.6

2016-06-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16009: Assignee: Tathagata Das (was: Apache Spark) > DataFrameRead.json(path) compatibility

[jira] [Assigned] (SPARK-15982) Harmonize the behavior of DataFrameReader.text/csv/json/parquet/orc

2016-06-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15982: Assignee: Tathagata Das (was: Apache Spark) > Harmonize the behavior of

[jira] [Assigned] (SPARK-15982) Harmonize the behavior of DataFrameReader.text/csv/json/parquet/orc

2016-06-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15982: Assignee: Apache Spark (was: Tathagata Das) > Harmonize the behavior of

[jira] [Commented] (SPARK-15982) Harmonize the behavior of DataFrameReader.text/csv/json/parquet/orc

2016-06-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335340#comment-15335340 ] Apache Spark commented on SPARK-15982: -- User 'tdas' has created a pull request for this issue:

[jira] [Assigned] (SPARK-16009) DataFrameRead.json(path) compatibility broken with Spark 1.6

2016-06-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16009: Assignee: Apache Spark (was: Tathagata Das) > DataFrameRead.json(path) compatibility

[jira] [Commented] (SPARK-16009) DataFrameRead.json(path) compatibility broken with Spark 1.6

2016-06-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335341#comment-15335341 ] Apache Spark commented on SPARK-16009: -- User 'tdas' has created a pull request for this issue:

[jira] [Commented] (SPARK-15982) Harmonize the behavior of DataFrameReader.text/csv/json/parquet/orc

2016-06-16 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335335#comment-15335335 ] Tathagata Das commented on SPARK-15982: --- I have started this. And this has because a complex

[jira] [Resolved] (SPARK-15908) Add varargs-type dropDuplicates() function in SparkR

2016-06-16 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman resolved SPARK-15908. --- Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request

[jira] [Updated] (SPARK-15908) Add varargs-type dropDuplicates() function in SparkR

2016-06-16 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman updated SPARK-15908: -- Assignee: Dongjoon Hyun > Add varargs-type dropDuplicates() function in SparkR

[jira] [Updated] (SPARK-15982) Harmonize the behavior of DataFrameReader.text/csv/json/parquet/orc

2016-06-16 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-15982: -- Description: Issues with current reader behavior. - `text()` without args returns an empty

[jira] [Commented] (SPARK-12177) Update KafkaDStreams to new Kafka 0.10 Consumer API

2016-06-16 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335332#comment-15335332 ] Cody Koeninger commented on SPARK-12177: The 0.8 consumer had mixed usage from different author's

[jira] [Issue Comment Deleted] (SPARK-12177) Update KafkaDStreams to new Kafka 0.10 Consumer API

2016-06-16 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cody Koeninger updated SPARK-12177: --- Comment: was deleted (was: Yes, that's related to that Kafka ticket, I noticed similar

[jira] [Assigned] (SPARK-16009) DataFrameRead.json(path) compatibility broken with Spark 1.6

2016-06-16 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das reassigned SPARK-16009: - Assignee: Tathagata Das > DataFrameRead.json(path) compatibility broken with Spark 1.6

[jira] [Created] (SPARK-16009) DataFrameRead.json(path) compatibility broken with Spark 1.6

2016-06-16 Thread Tathagata Das (JIRA)
Tathagata Das created SPARK-16009: - Summary: DataFrameRead.json(path) compatibility broken with Spark 1.6 Key: SPARK-16009 URL: https://issues.apache.org/jira/browse/SPARK-16009 Project: Spark

[jira] [Created] (SPARK-16008) ML Logistic Regression aggregator serializes unnecessary data

2016-06-16 Thread Seth Hendrickson (JIRA)
Seth Hendrickson created SPARK-16008: Summary: ML Logistic Regression aggregator serializes unnecessary data Key: SPARK-16008 URL: https://issues.apache.org/jira/browse/SPARK-16008 Project: Spark

[jira] [Updated] (SPARK-15982) Harmonize the behavior of DataFrameReader.text/csv/json/parquet/orc

2016-06-16 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-15982: -- Description: Issues with current reader behavior. - `text()` without args returns an empty

[jira] [Commented] (SPARK-16006) Attemping to write empty DataFrame with no fields throw non-intuitive exception

2016-06-16 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335324#comment-15335324 ] Dongjoon Hyun commented on SPARK-16006: --- I will fix this tonight~ Thank you for reporting this,

[jira] [Updated] (SPARK-15982) Harmonize the behavior of DataFrameReader.text/csv/json/parquet/orc

2016-06-16 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-15982: -- Summary: Harmonize the behavior of DataFrameReader.text/csv/json/parquet/orc (was:

[jira] [Commented] (SPARK-12177) Update KafkaDStreams to new Kafka 0.10 Consumer API

2016-06-16 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335311#comment-15335311 ] Cody Koeninger commented on SPARK-12177: [~jinx...@ebay.com] Replied by email, but just to put

[jira] [Updated] (SPARK-16006) Attemping to write empty DataFrame with no fields throw non-intuitive exception

2016-06-16 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-16006: -- Priority: Minor (was: Major) > Attemping to write empty DataFrame with no fields throw

[jira] [Updated] (SPARK-16006) Attemping to write empty DataFrame with no fields throw non-intuitive exception

2016-06-16 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-16006: -- Summary: Attemping to write empty DataFrame with no fields throw non-intuitive exception

[jira] [Updated] (SPARK-16006) Empty DataFrame with no fields created with spark.read.text() cannot be written as it has no fields

2016-06-16 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-16006: -- Description: Attempting to write an emptyDataFrame created with

[jira] [Commented] (SPARK-12177) Update KafkaDStreams to new Kafka 0.10 Consumer API

2016-06-16 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335297#comment-15335297 ] Cody Koeninger commented on SPARK-12177: Yes, that's related to that Kafka ticket, I noticed

[jira] [Updated] (SPARK-16006) Empty DataFrame with no fields created with spark.read.text() cannot be written as it has no fields

2016-06-16 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-16006: -- Description: Attempting to write an emptyDataFrame created with

[jira] [Updated] (SPARK-16007) Empty DataFrame created with spark.read.csv() does not respect user specified schema

2016-06-16 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-16007: -- Summary: Empty DataFrame created with spark.read.csv() does not respect user specified schema

[jira] [Updated] (SPARK-16006) Empty DataFrame with no fields created with spark.read.text() cannot be written as it has no fields

2016-06-16 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-16006: -- Summary: Empty DataFrame with no fields created with spark.read.text() cannot be written as it

[jira] [Updated] (SPARK-16007) Empty DataFrame created with spark.read.load() does not respect user specified schema

2016-06-16 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-16007: -- Priority: Minor (was: Major) > Empty DataFrame created with spark.read.load() does not

[jira] [Updated] (SPARK-16007) Empty DataFrame created with spark.read.load() does not respect user specified schema

2016-06-16 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-16007: -- Description: {{spark.schema(someSchema).csv().schema != someSchema}} The schema of the empty

[jira] [Resolved] (SPARK-15490) SparkR 2.0 QA: New R APIs and API docs for non-MLib changes

2016-06-16 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley resolved SPARK-15490. --- Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 13394

[jira] [Comment Edited] (SPARK-12177) Update KafkaDStreams to new Kafka 0.10 Consumer API

2016-06-16 Thread Jinxia Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335200#comment-15335200 ] Jinxia Liu edited comment on SPARK-12177 at 6/17/16 2:39 AM: -

[jira] [Created] (SPARK-16007) Empty DataFrame created with spark.read.load() does not respect user specified schema

2016-06-16 Thread Tathagata Das (JIRA)
Tathagata Das created SPARK-16007: - Summary: Empty DataFrame created with spark.read.load() does not respect user specified schema Key: SPARK-16007 URL: https://issues.apache.org/jira/browse/SPARK-16007

[jira] [Updated] (SPARK-16006) Empty DataFrame created with spark.read.text() cannot be written as it has no fields

2016-06-16 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-16006: -- Summary: Empty DataFrame created with spark.read.text() cannot be written as it has no fields

[jira] [Commented] (SPARK-15472) Add support for writing in `csv`, `json`, `text` formats in Structured Streaming

2016-06-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335238#comment-15335238 ] Apache Spark commented on SPARK-15472: -- User 'lw-lin' has created a pull request for this issue:

[jira] [Created] (SPARK-16006) Empty DataFrame with spark.read.text() cannot be written as it has no fields

2016-06-16 Thread Tathagata Das (JIRA)
Tathagata Das created SPARK-16006: - Summary: Empty DataFrame with spark.read.text() cannot be written as it has no fields Key: SPARK-16006 URL: https://issues.apache.org/jira/browse/SPARK-16006

[jira] [Comment Edited] (SPARK-12177) Update KafkaDStreams to new Kafka 0.10 Consumer API

2016-06-16 Thread Jinxia Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335215#comment-15335215 ] Jinxia Liu edited comment on SPARK-12177 at 6/17/16 2:03 AM: -

[jira] [Commented] (SPARK-15993) PySpark RuntimeConfig should be immutable

2016-06-16 Thread Jeff Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335218#comment-15335218 ] Jeff Zhang commented on SPARK-15993: RuntimeConfig in scala api is mutable, if it doesn't work in

[jira] [Comment Edited] (SPARK-12177) Update KafkaDStreams to new Kafka 0.10 Consumer API

2016-06-16 Thread Jinxia Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335200#comment-15335200 ] Jinxia Liu edited comment on SPARK-12177 at 6/17/16 2:03 AM: -

[jira] [Commented] (SPARK-12177) Update KafkaDStreams to new Kafka 0.10 Consumer API

2016-06-16 Thread Jinxia Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335215#comment-15335215 ] Jinxia Liu commented on SPARK-12177: Hi Cody, should the log usage be unified in the kafka streaming

[jira] [Commented] (SPARK-12177) Update KafkaDStreams to new Kafka 0.10 Consumer API

2016-06-16 Thread Jinxia Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335200#comment-15335200 ] Jinxia Liu commented on SPARK-12177: Hi Cody, I did some tests today using your connector, and

[jira] [Commented] (SPARK-13288) [1.6.0] Memory leak in Spark streaming

2016-06-16 Thread roberto hashioka (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335184#comment-15335184 ] roberto hashioka commented on SPARK-13288: -- I'm having the same issue. I'll try with Spark 1.5.1

[jira] [Commented] (SPARK-14048) Aggregation operations on structs fail when the structs have fields with special characters

2016-06-16 Thread Sean Zhong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335174#comment-15335174 ] Sean Zhong commented on SPARK-14048: [~simeons] You can use {{sqlContext.sql("query")}} instead of

[jira] [Commented] (SPARK-15892) Incorrectly merged AFTAggregator with zero total count

2016-06-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335171#comment-15335171 ] Apache Spark commented on SPARK-15892: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-15892) Incorrectly merged AFTAggregator with zero total count

2016-06-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15892: Assignee: Hyukjin Kwon (was: Apache Spark) > Incorrectly merged AFTAggregator with zero

[jira] [Assigned] (SPARK-15892) Incorrectly merged AFTAggregator with zero total count

2016-06-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15892: Assignee: Apache Spark (was: Hyukjin Kwon) > Incorrectly merged AFTAggregator with zero

[jira] [Commented] (SPARK-14351) Optimize ImpurityAggregator for decision trees

2016-06-16 Thread Manoj Kumar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335165#comment-15335165 ] Manoj Kumar commented on SPARK-14351: - OK, so here are some benchmarks that validate your claims

[jira] [Resolved] (SPARK-15782) --packages doesn't work with the spark-shell

2016-06-16 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-15782. Resolution: Fixed > --packages doesn't work with the spark-shell >

[jira] [Comment Edited] (SPARK-15581) MLlib 2.1 Roadmap

2016-06-16 Thread Alexander Ulanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15325377#comment-15325377 ] Alexander Ulanov edited comment on SPARK-15581 at 6/17/16 1:18 AM: --- I

[jira] [Comment Edited] (SPARK-15581) MLlib 2.1 Roadmap

2016-06-16 Thread Alexander Ulanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15325377#comment-15325377 ] Alexander Ulanov edited comment on SPARK-15581 at 6/17/16 1:18 AM: --- I

[jira] [Assigned] (SPARK-15973) Fix GroupedData Documentation

2016-06-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15973: Assignee: (was: Apache Spark) > Fix GroupedData Documentation >

[jira] [Commented] (SPARK-15973) Fix GroupedData Documentation

2016-06-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335114#comment-15335114 ] Apache Spark commented on SPARK-15973: -- User 'josh-howes' has created a pull request for this issue:

[jira] [Assigned] (SPARK-15973) Fix GroupedData Documentation

2016-06-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15973: Assignee: Apache Spark > Fix GroupedData Documentation > - >

[jira] [Commented] (SPARK-15822) segmentation violation in o.a.s.unsafe.types.UTF8String

2016-06-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335099#comment-15335099 ] Apache Spark commented on SPARK-15822: -- User 'hvanhovell' has created a pull request for this issue:

[jira] [Resolved] (SPARK-15608) Add document for ML IsotonicRegression

2016-06-16 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanbo Liang resolved SPARK-15608. - Resolution: Fixed Fix Version/s: 2.0.0 > Add document for ML IsotonicRegression >

[jira] [Updated] (SPARK-15608) Add document for ML IsotonicRegression

2016-06-16 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanbo Liang updated SPARK-15608: Assignee: Weichen Xu > Add document for ML IsotonicRegression >

[jira] [Commented] (SPARK-15343) NoClassDefFoundError when initializing Spark with YARN

2016-06-16 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335055#comment-15335055 ] Marcelo Vanzin commented on SPARK-15343: Yes, Spark 2.0 updated the version of Jersey, because

[jira] [Commented] (SPARK-14995) Add "since" tag in Roxygen documentation for SparkR API methods

2016-06-16 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335053#comment-15335053 ] Dongjoon Hyun commented on SPARK-14995: --- Hi, All. If it's okay, I'll make a PR for this tonight.

[jira] [Commented] (SPARK-15991) After SparkSession has been created, setting hadoop conf through sparkSession.sparkContext.hadoopConfiguration does not affect hadoop conf used by the SparkSession

2016-06-16 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335052#comment-15335052 ] Yin Huai commented on SPARK-15991: -- In the release note, we need to mention that if hive-site.xml is in

[jira] [Commented] (SPARK-15343) NoClassDefFoundError when initializing Spark with YARN

2016-06-16 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335039#comment-15335039 ] Saisai Shao commented on SPARK-15343: - If timeline is enabled, YarnClient will also post some events

[jira] [Resolved] (SPARK-15991) After SparkSession has been created, setting hadoop conf through sparkSession.sparkContext.hadoopConfiguration does not affect hadoop conf used by the SparkSession

2016-06-16 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-15991. -- Resolution: Fixed Fix Version/s: 2.0.0 > After SparkSession has been created, setting

[jira] [Commented] (SPARK-15925) Replaces registerTempTable with createOrReplaceTempView in SparkR

2016-06-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335026#comment-15335026 ] Apache Spark commented on SPARK-15925: -- User 'felixcheung' has created a pull request for this

[jira] [Resolved] (SPARK-15966) Fix markdown for Spark Monitoring

2016-06-16 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-15966. - Resolution: Fixed Assignee: Dhruve Ashar Fix Version/s: 2.0.0 > Fix markdown for

[jira] [Assigned] (SPARK-16005) Add `randomSplit` to SparkR

2016-06-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16005: Assignee: Apache Spark > Add `randomSplit` to SparkR > --- > >

[jira] [Assigned] (SPARK-16005) Add `randomSplit` to SparkR

2016-06-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16005: Assignee: (was: Apache Spark) > Add `randomSplit` to SparkR >

[jira] [Commented] (SPARK-16005) Add `randomSplit` to SparkR

2016-06-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15334976#comment-15334976 ] Apache Spark commented on SPARK-16005: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Created] (SPARK-16005) Add `randomSplit` to SparkR

2016-06-16 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-16005: - Summary: Add `randomSplit` to SparkR Key: SPARK-16005 URL: https://issues.apache.org/jira/browse/SPARK-16005 Project: Spark Issue Type: Bug

[jira] [Assigned] (SPARK-16004) Improve CatalogTable information

2016-06-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16004: Assignee: Apache Spark > Improve CatalogTable information >

[jira] [Assigned] (SPARK-16004) Improve CatalogTable information

2016-06-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16004: Assignee: (was: Apache Spark) > Improve CatalogTable information >

[jira] [Commented] (SPARK-16004) Improve CatalogTable information

2016-06-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15334956#comment-15334956 ] Apache Spark commented on SPARK-16004: -- User 'bomeng' has created a pull request for this issue:

[jira] [Created] (SPARK-16004) Improve CatalogTable information

2016-06-16 Thread Bo Meng (JIRA)
Bo Meng created SPARK-16004: --- Summary: Improve CatalogTable information Key: SPARK-16004 URL: https://issues.apache.org/jira/browse/SPARK-16004 Project: Spark Issue Type: Bug Components:

[jira] [Updated] (SPARK-15999) Wrong/Missing information for Spark UI/REST port

2016-06-16 Thread Faisal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Faisal updated SPARK-15999: --- Description: *Spark Monitoring documentation* https://spark.apache.org/docs/1.5.0/monitoring.html {quote}

[jira] [Updated] (SPARK-15999) Wrong/Missing information for Spark UI/REST port

2016-06-16 Thread Faisal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Faisal updated SPARK-15999: --- Description: *Spark Monitoring documentation* https://spark.apache.org/docs/1.5.0/monitoring.html {quote}

[jira] [Reopened] (SPARK-15999) Wrong/Missing information for Spark UI/REST port

2016-06-16 Thread Faisal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Faisal reopened SPARK-15999: > Wrong/Missing information for Spark UI/REST port > > >

[jira] [Comment Edited] (SPARK-15999) Wrong/Missing information for Spark UI/REST port

2016-06-16 Thread Faisal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15334895#comment-15334895 ] Faisal edited comment on SPARK-15999 at 6/16/16 11:03 PM: -- Appreciate your

[jira] [Commented] (SPARK-15999) Wrong/Missing information for Spark UI/REST port

2016-06-16 Thread Faisal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15334895#comment-15334895 ] Faisal commented on SPARK-15999: Appreciate your prompt response but seems like it was never tested that

[jira] [Updated] (SPARK-16003) SerializationDebugger run into infinite loop

2016-06-16 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-16003: --- Description: This is observed while debugging https://issues.apache.org/jira/browse/SPARK-15811

[jira] [Updated] (SPARK-16003) SerializationDebugger run into infinite loop

2016-06-16 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-16003: --- Description: This is observed while debugging https://issues.apache.org/jira/browse/SPARK-15811

  1   2   3   4   >