[jira] [Commented] (SPARK-23402) Dataset write method not working as expected for postgresql database

2018-02-13 Thread kevin yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363584#comment-16363584 ] kevin yu commented on SPARK-23402: -- Thanks, I will install the 9.5.8, and try again. Sent from my

[jira] [Comment Edited] (SPARK-23402) Dataset write method not working as expected for postgresql database

2018-02-13 Thread Pallapothu Jyothi Swaroop (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363555#comment-16363555 ] Pallapothu Jyothi Swaroop edited comment on SPARK-23402 at 2/14/18 6:48 AM:

[jira] [Commented] (SPARK-23402) Dataset write method not working as expected for postgresql database

2018-02-13 Thread Pallapothu Jyothi Swaroop (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363555#comment-16363555 ] Pallapothu Jyothi Swaroop commented on SPARK-23402: --- Thanks for checking again. I

[jira] [Commented] (SPARK-23402) Dataset write method not working as expected for postgresql database

2018-02-13 Thread kevin yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363539#comment-16363539 ] kevin yu commented on SPARK-23402: -- Yes, I create empty table (emptytable) in database (mydb) in the

[jira] [Updated] (SPARK-23368) Avoid unnecessary Exchange or Sort after projection

2018-02-13 Thread Maryann Xue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maryann Xue updated SPARK-23368: Summary: Avoid unnecessary Exchange or Sort after projection (was: OutputOrdering and

[jira] [Commented] (SPARK-23402) Dataset write method not working as expected for postgresql database

2018-02-13 Thread Pallapothu Jyothi Swaroop (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363512#comment-16363512 ] Pallapothu Jyothi Swaroop commented on SPARK-23402: --- [~kevinyu98] Did you create table

[jira] [Created] (SPARK-23420) Datasource loading not handling paths with regex chars.

2018-02-13 Thread Mitchell (JIRA)
Mitchell created SPARK-23420: Summary: Datasource loading not handling paths with regex chars. Key: SPARK-23420 URL: https://issues.apache.org/jira/browse/SPARK-23420 Project: Spark Issue Type:

[jira] [Updated] (SPARK-23230) When hive.default.fileformat is other kinds of file types, create textfile table cause a serde error

2018-02-13 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23230: Fix Version/s: 2.2.2 > When hive.default.fileformat is other kinds of file types, create textfile > table

[jira] [Resolved] (SPARK-23399) Register a task completion listener first for OrcColumnarBatchReader

2018-02-13 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-23399. - Resolution: Fixed Fix Version/s: 2.3.1 Issue resolved by pull request 20590

[jira] [Assigned] (SPARK-23399) Register a task completion listener first for OrcColumnarBatchReader

2018-02-13 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-23399: --- Assignee: Dongjoon Hyun > Register a task completion listener first for

[jira] [Commented] (SPARK-23419) data source v2 write path should re-throw interruption exceptions directly

2018-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363385#comment-16363385 ] Apache Spark commented on SPARK-23419: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Assigned] (SPARK-23419) data source v2 write path should re-throw interruption exceptions directly

2018-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23419: Assignee: Apache Spark (was: Wenchen Fan) > data source v2 write path should re-throw

[jira] [Assigned] (SPARK-23419) data source v2 write path should re-throw interruption exceptions directly

2018-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23419: Assignee: Wenchen Fan (was: Apache Spark) > data source v2 write path should re-throw

[jira] [Commented] (SPARK-23416) flaky test: org.apache.spark.sql.kafka010.KafkaSourceStressForDontFailOnDataLossSuite.stress test for failOnDataLoss=false

2018-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363386#comment-16363386 ] Apache Spark commented on SPARK-23416: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Updated] (SPARK-23419) data source v2 write path should re-throw interruption exceptions directly

2018-02-13 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-23419: Summary: data source v2 write path should re-throw interruption exceptions directly (was: data

[jira] [Commented] (SPARK-23377) Bucketizer with multiple columns persistence bug

2018-02-13 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363367#comment-16363367 ] Liang-Chi Hsieh commented on SPARK-23377: - I agree with what [~mlnick] said. > Bucketizer with

[jira] [Updated] (SPARK-23419) data source v2 write path should re-throw fatal errors directly

2018-02-13 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-23419: Summary: data source v2 write path should re-throw fatal errors directly (was: data source v2

[jira] [Created] (SPARK-23419) data source v2 write path should re-throw FetchFailedException directly

2018-02-13 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-23419: --- Summary: data source v2 write path should re-throw FetchFailedException directly Key: SPARK-23419 URL: https://issues.apache.org/jira/browse/SPARK-23419 Project: Spark

[jira] [Commented] (SPARK-12140) Support Streaming UI in HistoryServer

2018-02-13 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363300#comment-16363300 ] Saisai Shao commented on SPARK-12140: - [~gschiavon] the community has concern about supporting

[jira] [Commented] (SPARK-23308) ignoreCorruptFiles should not ignore retryable IOException

2018-02-13 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-23308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363217#comment-16363217 ] Márcio Furlani Carmona commented on SPARK-23308: {quote}if your input stream is doing

[jira] [Resolved] (SPARK-23235) Add executor Threaddump to api

2018-02-13 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-23235. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20474

[jira] [Assigned] (SPARK-23235) Add executor Threaddump to api

2018-02-13 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-23235: Assignee: Attila Zsolt Piros > Add executor Threaddump to api >

[jira] [Commented] (SPARK-23351) checkpoint corruption in long running application

2018-02-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363133#comment-16363133 ] Sean Owen commented on SPARK-23351: --- [~davidahern] for the record, it's usually the opposite. Vendor

[jira] [Assigned] (SPARK-23365) DynamicAllocation with failure in straggler task can lead to a hung spark job

2018-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23365: Assignee: (was: Apache Spark) > DynamicAllocation with failure in straggler task can

[jira] [Assigned] (SPARK-23365) DynamicAllocation with failure in straggler task can lead to a hung spark job

2018-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23365: Assignee: Apache Spark > DynamicAllocation with failure in straggler task can lead to a

[jira] [Commented] (SPARK-23365) DynamicAllocation with failure in straggler task can lead to a hung spark job

2018-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363130#comment-16363130 ] Apache Spark commented on SPARK-23365: -- User 'squito' has created a pull request for this issue:

[jira] [Assigned] (SPARK-23418) DataSourceV2 should not allow userSpecifiedSchema without ReadSupportWithSchema

2018-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23418: Assignee: Apache Spark > DataSourceV2 should not allow userSpecifiedSchema without >

[jira] [Assigned] (SPARK-23418) DataSourceV2 should not allow userSpecifiedSchema without ReadSupportWithSchema

2018-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23418: Assignee: (was: Apache Spark) > DataSourceV2 should not allow userSpecifiedSchema

[jira] [Commented] (SPARK-23418) DataSourceV2 should not allow userSpecifiedSchema without ReadSupportWithSchema

2018-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363127#comment-16363127 ] Apache Spark commented on SPARK-23418: -- User 'rdblue' has created a pull request for this issue:

[jira] [Created] (SPARK-23418) DataSourceV2 should not allow userSpecifiedSchema without ReadSupportWithSchema

2018-02-13 Thread Ryan Blue (JIRA)
Ryan Blue created SPARK-23418: - Summary: DataSourceV2 should not allow userSpecifiedSchema without ReadSupportWithSchema Key: SPARK-23418 URL: https://issues.apache.org/jira/browse/SPARK-23418 Project:

[jira] [Commented] (SPARK-23351) checkpoint corruption in long running application

2018-02-13 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363115#comment-16363115 ] Shixiong Zhu commented on SPARK-23351: -- [~davidahern] It's better to ask the vendor for support.

[jira] [Commented] (SPARK-23402) Dataset write method not working as expected for postgresql database

2018-02-13 Thread kevin yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363110#comment-16363110 ] kevin yu commented on SPARK-23402: -- I just tried with  PostgreSQL 9.5.6 on x86_64-pc-linux-gnu with

[jira] [Commented] (SPARK-23411) Deprecate SparkContext.getRDDStorageInfo

2018-02-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363037#comment-16363037 ] Marcelo Vanzin commented on SPARK-23411: Argh, copied the wrong method name. Fixed. > Deprecate

[jira] [Updated] (SPARK-23411) Deprecate SparkContext.getRDDStorageInfo

2018-02-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-23411: --- Summary: Deprecate SparkContext.getRDDStorageInfo (was: Deprecate

[jira] [Commented] (SPARK-23414) Plotting using matplotlib in MLlib pyspark

2018-02-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363029#comment-16363029 ] Sean Owen commented on SPARK-23414: --- matplotlib doesn't interact with Spark, so issues with using it

[jira] [Commented] (SPARK-23414) Plotting using matplotlib in MLlib pyspark

2018-02-13 Thread Waleed Esmail (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363025#comment-16363025 ] Waleed Esmail commented on SPARK-23414: --- I am sorry, I didn't get it, what do you mean by

[jira] [Created] (SPARK-23417) pyspark tests give wrong sbt instructions

2018-02-13 Thread Jose Torres (JIRA)
Jose Torres created SPARK-23417: --- Summary: pyspark tests give wrong sbt instructions Key: SPARK-23417 URL: https://issues.apache.org/jira/browse/SPARK-23417 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-23400) Add the extra constructors for ScalaUDF

2018-02-13 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-23400: - Fix Version/s: 2.4.0 > Add the extra constructors for ScalaUDF >

[jira] [Resolved] (SPARK-23400) Add the extra constructors for ScalaUDF

2018-02-13 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-23400. -- Resolution: Fixed Fix Version/s: 2.3.1 Issue resolved by pull request 20591

[jira] [Commented] (SPARK-23416) flaky test: org.apache.spark.sql.kafka010.KafkaSourceStressForDontFailOnDataLossSuite.stress test for failOnDataLoss=false

2018-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362950#comment-16362950 ] Apache Spark commented on SPARK-23416: -- User 'jose-torres' has created a pull request for this

[jira] [Assigned] (SPARK-23416) flaky test: org.apache.spark.sql.kafka010.KafkaSourceStressForDontFailOnDataLossSuite.stress test for failOnDataLoss=false

2018-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23416: Assignee: Apache Spark > flaky test: >

[jira] [Assigned] (SPARK-23416) flaky test: org.apache.spark.sql.kafka010.KafkaSourceStressForDontFailOnDataLossSuite.stress test for failOnDataLoss=false

2018-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23416: Assignee: (was: Apache Spark) > flaky test: >

[jira] [Comment Edited] (SPARK-23416) flaky test: org.apache.spark.sql.kafka010.KafkaSourceStressForDontFailOnDataLossSuite.stress test for failOnDataLoss=false

2018-02-13 Thread Jose Torres (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362941#comment-16362941 ] Jose Torres edited comment on SPARK-23416 at 2/13/18 7:48 PM: -- I think I see

[jira] [Commented] (SPARK-23416) flaky test: org.apache.spark.sql.kafka010.KafkaSourceStressForDontFailOnDataLossSuite.stress test for failOnDataLoss=false

2018-02-13 Thread Jose Torres (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362941#comment-16362941 ] Jose Torres commented on SPARK-23416: - I think I see the problem. * StreamExecution.stop() works by

[jira] [Commented] (SPARK-23416) flaky test: org.apache.spark.sql.kafka010.KafkaSourceStressForDontFailOnDataLossSuite.stress test for failOnDataLoss=false

2018-02-13 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362920#comment-16362920 ] Marco Gaido commented on SPARK-23416: - I see this failing also with this stacktrace: {code:java}

[jira] [Commented] (SPARK-23416) flaky test: org.apache.spark.sql.kafka010.KafkaSourceStressForDontFailOnDataLossSuite.stress test for failOnDataLoss=false

2018-02-13 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362913#comment-16362913 ] Dongjoon Hyun commented on SPARK-23416: --- Thank you for filing this! > flaky test: >

[jira] [Resolved] (SPARK-23154) Document backwards compatibility guarantees for ML persistence

2018-02-13 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley resolved SPARK-23154. --- Resolution: Fixed Fix Version/s: 2.4.0 2.3.1 Resolved via

[jira] [Commented] (SPARK-23344) Add KMeans distanceMeasure param to PySpark

2018-02-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362902#comment-16362902 ] Sean Owen commented on SPARK-23344: --- Nah, I think the regulars have different views on this, which

[jira] [Updated] (SPARK-23159) Update Cloudpickle to match version 0.4.3

2018-02-13 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-23159: - Description: Update PySpark's version of Cloudpickle to match version 0.4.3.  The reasons for

[jira] [Updated] (SPARK-23159) Update Cloudpickle to match version 0.4.3

2018-02-13 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-23159: - Summary: Update Cloudpickle to match version 0.4.3 (was: Update Cloudpickle to match version

[jira] [Created] (SPARK-23416) flaky test: org.apache.spark.sql.kafka010.KafkaSourceStressForDontFailOnDataLossSuite.stress test for failOnDataLoss=false

2018-02-13 Thread Jose Torres (JIRA)
Jose Torres created SPARK-23416: --- Summary: flaky test: org.apache.spark.sql.kafka010.KafkaSourceStressForDontFailOnDataLossSuite.stress test for failOnDataLoss=false Key: SPARK-23416 URL:

[jira] [Created] (SPARK-23415) BufferHolderSparkSubmitSuite is flaky

2018-02-13 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-23415: - Summary: BufferHolderSparkSubmitSuite is flaky Key: SPARK-23415 URL: https://issues.apache.org/jira/browse/SPARK-23415 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-23410) Unable to read jsons in charset different from UTF-8

2018-02-13 Thread Maxim Gekk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-23410: --- Shepherd: Herman van Hovell > Unable to read jsons in charset different from UTF-8 >

[jira] [Assigned] (SPARK-23413) Sorting tasks by Host / Executor ID on the Stage page does not work

2018-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23413: Assignee: Apache Spark > Sorting tasks by Host / Executor ID on the Stage page does not

[jira] [Commented] (SPARK-23413) Sorting tasks by Host / Executor ID on the Stage page does not work

2018-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362830#comment-16362830 ] Apache Spark commented on SPARK-23413: -- User 'attilapiros' has created a pull request for this

[jira] [Assigned] (SPARK-23413) Sorting tasks by Host / Executor ID on the Stage page does not work

2018-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23413: Assignee: (was: Apache Spark) > Sorting tasks by Host / Executor ID on the Stage page

[jira] [Commented] (SPARK-23344) Add KMeans distanceMeasure param to PySpark

2018-02-13 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362774#comment-16362774 ] Marco Gaido commented on SPARK-23344: - I see. It would be good indeed to decide in the community a

[jira] [Commented] (SPARK-23344) Add KMeans distanceMeasure param to PySpark

2018-02-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362760#comment-16362760 ] Sean Owen commented on SPARK-23344: --- My general rule of thumb is: how likely is it you would resolve

[jira] [Commented] (SPARK-23344) Add KMeans distanceMeasure param to PySpark

2018-02-13 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362758#comment-16362758 ] Marco Gaido commented on SPARK-23344: - [~srowen] I did it this way because I always say doing so. Not

[jira] [Updated] (SPARK-23292) python tests related to pandas are skipped with python 2

2018-02-13 Thread Sameer Agarwal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sameer Agarwal updated SPARK-23292: --- Summary: python tests related to pandas are skipped with python 2 (was: python tests

[jira] [Resolved] (SPARK-23217) Add cosine distance measure to ClusteringEvaluator

2018-02-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-23217. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20396

[jira] [Assigned] (SPARK-23217) Add cosine distance measure to ClusteringEvaluator

2018-02-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-23217: - Assignee: Marco Gaido > Add cosine distance measure to ClusteringEvaluator >

[jira] [Resolved] (SPARK-23392) Add some test case for images feature

2018-02-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-23392. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20583

[jira] [Assigned] (SPARK-23392) Add some test case for images feature

2018-02-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-23392: - Assignee: xubo245 > Add some test case for images feature >

[jira] [Commented] (SPARK-23388) Support for Parquet Binary DecimalType in VectorizedColumnReader

2018-02-13 Thread Sameer Agarwal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362730#comment-16362730 ] Sameer Agarwal commented on SPARK-23388: yes, I agree > Support for Parquet Binary DecimalType

[jira] [Assigned] (SPARK-23382) Spark Streaming ui about the contents of the form need to have hidden and show features, when the table records very much.

2018-02-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-23382: - Assignee: guoxiaolongzte > Spark Streaming ui about the contents of the form need to have

[jira] [Resolved] (SPARK-23382) Spark Streaming ui about the contents of the form need to have hidden and show features, when the table records very much.

2018-02-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-23382. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20570

[jira] [Updated] (SPARK-23340) Upgrade Apache ORC to 1.4.3

2018-02-13 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-23340: -- Summary: Upgrade Apache ORC to 1.4.3 (was: Empty float/double array columns in ORC file

[jira] [Resolved] (SPARK-23414) Plotting using matplotlib in MLlib pyspark

2018-02-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-23414. --- Resolution: Invalid Questions should go to the mailing list. Using matplotlib would be out of scope

[jira] [Created] (SPARK-23414) Plotting using matplotlib in MLlib pyspark

2018-02-13 Thread Waleed Esmail (JIRA)
Waleed Esmail created SPARK-23414: - Summary: Plotting using matplotlib in MLlib pyspark Key: SPARK-23414 URL: https://issues.apache.org/jira/browse/SPARK-23414 Project: Spark Issue Type:

[jira] [Commented] (SPARK-23411) Deprecate SparkContext.getExecutorStorageStatus

2018-02-13 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362597#comment-16362597 ] Marco Gaido commented on SPARK-23411: - I think this method was removed in SPARK-20659. So I think

[jira] [Resolved] (SPARK-23053) taskBinarySerialization and task partitions calculate in DagScheduler.submitMissingTasks should keep the same RDD checkpoint status

2018-02-13 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-23053. -- Resolution: Fixed > taskBinarySerialization and task partitions calculate in >

[jira] [Assigned] (SPARK-23053) taskBinarySerialization and task partitions calculate in DagScheduler.submitMissingTasks should keep the same RDD checkpoint status

2018-02-13 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-23053: Assignee: huangtengfei > taskBinarySerialization and task partitions calculate in >

[jira] [Commented] (SPARK-23053) taskBinarySerialization and task partitions calculate in DagScheduler.submitMissingTasks should keep the same RDD checkpoint status

2018-02-13 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362556#comment-16362556 ] Imran Rashid commented on SPARK-23053: -- Fixed by https://github.com/apache/spark/pull/20244 I set

[jira] [Updated] (SPARK-23053) taskBinarySerialization and task partitions calculate in DagScheduler.submitMissingTasks should keep the same RDD checkpoint status

2018-02-13 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-23053: - Fix Version/s: 2.4.0 2.3.1 2.2.2 > taskBinarySerialization

[jira] [Assigned] (SPARK-23189) reflect stage level blacklisting on executor tab

2018-02-13 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-23189: Assignee: Attila Zsolt Piros > reflect stage level blacklisting on executor tab >

[jira] [Resolved] (SPARK-23189) reflect stage level blacklisting on executor tab

2018-02-13 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-23189. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20408

[jira] [Updated] (SPARK-23413) Sorting tasks by Host / Executor ID on the Stage page does not work

2018-02-13 Thread Attila Zsolt Piros (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Zsolt Piros updated SPARK-23413: --- Description: Sorting tasks by Host / Executor ID throws exceptions:  {code}

[jira] [Commented] (SPARK-23413) Sorting tasks by Host / Executor ID on the Stage page does not work

2018-02-13 Thread Attila Zsolt Piros (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362513#comment-16362513 ] Attila Zsolt Piros commented on SPARK-23413: I am working on that. > Sorting tasks by Host /

[jira] [Created] (SPARK-23413) Sorting tasks by Host / Executor ID on the Stage page does not work

2018-02-13 Thread Attila Zsolt Piros (JIRA)
Attila Zsolt Piros created SPARK-23413: -- Summary: Sorting tasks by Host / Executor ID on the Stage page does not work Key: SPARK-23413 URL: https://issues.apache.org/jira/browse/SPARK-23413

[jira] [Commented] (SPARK-23412) Add cosine distance measure to BisectingKMeans

2018-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362493#comment-16362493 ] Apache Spark commented on SPARK-23412: -- User 'mgaido91' has created a pull request for this issue:

[jira] [Assigned] (SPARK-23412) Add cosine distance measure to BisectingKMeans

2018-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23412: Assignee: Apache Spark > Add cosine distance measure to BisectingKMeans >

[jira] [Assigned] (SPARK-23412) Add cosine distance measure to BisectingKMeans

2018-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23412: Assignee: (was: Apache Spark) > Add cosine distance measure to BisectingKMeans >

[jira] [Created] (SPARK-23412) Add cosine distance measure to BisectingKMeans

2018-02-13 Thread Marco Gaido (JIRA)
Marco Gaido created SPARK-23412: --- Summary: Add cosine distance measure to BisectingKMeans Key: SPARK-23412 URL: https://issues.apache.org/jira/browse/SPARK-23412 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-20659) Remove StorageStatus, or make it private.

2018-02-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-20659: -- Assignee: Attila Zsolt Piros > Remove StorageStatus, or make it private. >

[jira] [Resolved] (SPARK-20659) Remove StorageStatus, or make it private.

2018-02-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-20659. Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20546

[jira] [Updated] (SPARK-23411) Deprecate SparkContext.getExecutorStorageStatus

2018-02-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-23411: --- Issue Type: Improvement (was: Bug) > Deprecate SparkContext.getExecutorStorageStatus >

[jira] [Created] (SPARK-23411) Deprecate SparkContext.getExecutorStorageStatus

2018-02-13 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-23411: -- Summary: Deprecate SparkContext.getExecutorStorageStatus Key: SPARK-23411 URL: https://issues.apache.org/jira/browse/SPARK-23411 Project: Spark Issue

[jira] [Commented] (SPARK-21501) Spark shuffle index cache size should be memory based

2018-02-13 Thread Xun REN (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362424#comment-16362424 ] Xun REN commented on SPARK-21501: - Hi guys, Could you tell me how to figure out how many memory the NM

[jira] [Updated] (SPARK-21232) New built-in SQL function - Data_Type

2018-02-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-21232: -- Fix Version/s: (was: 2.2.2) (was: 2.3.0) > New built-in SQL function -

[jira] [Updated] (SPARK-23395) Add an option to return an empty DataFrame from an RDD generated by a Hadoop file when there are no usable paths

2018-02-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-23395: -- Target Version/s: (was: 2.2.0, 2.2.1) > Add an option to return an empty DataFrame from an RDD

[jira] [Created] (SPARK-23410) Unable to read jsons in charset different from UTF-8

2018-02-13 Thread Maxim Gekk (JIRA)
Maxim Gekk created SPARK-23410: -- Summary: Unable to read jsons in charset different from UTF-8 Key: SPARK-23410 URL: https://issues.apache.org/jira/browse/SPARK-23410 Project: Spark Issue Type:

[jira] [Created] (SPARK-23409) RandomForest/DecisionTree (syntactic) pruning of redundant subtrees

2018-02-13 Thread Alessandro Solimando (JIRA)
Alessandro Solimando created SPARK-23409: Summary: RandomForest/DecisionTree (syntactic) pruning of redundant subtrees Key: SPARK-23409 URL: https://issues.apache.org/jira/browse/SPARK-23409

[jira] [Created] (SPARK-23408) Flaky test: StreamingOuterJoinSuite.left outer early state exclusion on right

2018-02-13 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-23408: -- Summary: Flaky test: StreamingOuterJoinSuite.left outer early state exclusion on right Key: SPARK-23408 URL: https://issues.apache.org/jira/browse/SPARK-23408

[jira] [Resolved] (SPARK-23364) 'desc table' command in spark-sql add column head display

2018-02-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-23364. --- Resolution: Not A Problem > 'desc table' command in spark-sql add column head display >

[jira] [Assigned] (SPARK-23407) add a config to try to inline all mutable states during codegen

2018-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23407: Assignee: Wenchen Fan (was: Apache Spark) > add a config to try to inline all mutable

[jira] [Assigned] (SPARK-23407) add a config to try to inline all mutable states during codegen

2018-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23407: Assignee: Apache Spark (was: Wenchen Fan) > add a config to try to inline all mutable

[jira] [Commented] (SPARK-23407) add a config to try to inline all mutable states during codegen

2018-02-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362218#comment-16362218 ] Apache Spark commented on SPARK-23407: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Commented] (SPARK-12140) Support Streaming UI in HistoryServer

2018-02-13 Thread German Schiavon Matteo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362216#comment-16362216 ] German Schiavon Matteo commented on SPARK-12140: Hi guys, is there any progress about

[jira] [Resolved] (SPARK-23396) Spark HistoryServer will OMM if the event log is big

2018-02-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-23396. --- Resolution: Not A Problem > Spark HistoryServer will OMM if the event log is big >

  1   2   >