[jira] [Updated] (SPARK-23265) Update multi-column error handling logic in QuantileDiscretizer

2018-01-29 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath updated SPARK-23265: --- Description: SPARK-22397 added support for multiple columns to {{QuantileDiscretizer}}. If

[jira] [Commented] (SPARK-23265) Update multi-column error handling logic in QuantileDiscretizer

2018-01-29 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344604#comment-16344604 ] Nick Pentreath commented on SPARK-23265: cc [~huaxing]  > Update multi-column error handling

[jira] [Updated] (SPARK-23265) Update multi-column error handling logic in QuantileDiscretizer

2018-01-29 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath updated SPARK-23265: --- Issue Type: Improvement (was: Documentation) > Update multi-column error handling logic in

[jira] [Created] (SPARK-23265) Update multi-column error handling logic in QuantileDiscretizer

2018-01-29 Thread Nick Pentreath (JIRA)
Nick Pentreath created SPARK-23265: -- Summary: Update multi-column error handling logic in QuantileDiscretizer Key: SPARK-23265 URL: https://issues.apache.org/jira/browse/SPARK-23265 Project: Spark

[jira] [Updated] (SPARK-23265) Update multi-column error handling logic in QuantileDiscretizer

2018-01-29 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath updated SPARK-23265: --- Description: SPARK-22397 added support for multiple columns to {{QuantileDiscretizer}}. If

[jira] [Resolved] (SPARK-23138) Add user guide example for multiclass logistic regression summary

2018-01-29 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath resolved SPARK-23138. Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 20332

[jira] [Assigned] (SPARK-23138) Add user guide example for multiclass logistic regression summary

2018-01-29 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath reassigned SPARK-23138: -- Assignee: Seth Hendrickson > Add user guide example for multiclass logistic

[jira] [Commented] (SPARK-20928) SPIP: Continuous Processing Mode for Structured Streaming

2018-01-29 Thread liweisheng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344565#comment-16344565 ] liweisheng commented on SPARK-20928: What about introducing a new way of non-block shuffle,I mean

[jira] [Assigned] (SPARK-23264) Support interval values without INTERVAL clauses

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23264: Assignee: (was: Apache Spark) > Support interval values without INTERVAL clauses >

[jira] [Assigned] (SPARK-23264) Support interval values without INTERVAL clauses

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23264: Assignee: Apache Spark > Support interval values without INTERVAL clauses >

[jira] [Commented] (SPARK-23264) Support interval values without INTERVAL clauses

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344561#comment-16344561 ] Apache Spark commented on SPARK-23264: -- User 'maropu' has created a pull request for this issue:

[jira] [Resolved] (SPARK-23157) withColumn fails for a column that is a result of mapped DataSet

2018-01-29 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-23157. - Resolution: Invalid > withColumn fails for a column that is a result of mapped DataSet >

[jira] [Created] (SPARK-23264) Support interval values without INTERVAL clauses

2018-01-29 Thread Takeshi Yamamuro (JIRA)
Takeshi Yamamuro created SPARK-23264: Summary: Support interval values without INTERVAL clauses Key: SPARK-23264 URL: https://issues.apache.org/jira/browse/SPARK-23264 Project: Spark

[jira] [Commented] (SPARK-23174) Fix pep8 to latest official version

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344553#comment-16344553 ] Apache Spark commented on SPARK-23174: -- User 'ueshin' has created a pull request for this issue:

[jira] [Assigned] (SPARK-23222) Flaky test: DataFrameRangeSuite

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23222: Assignee: Apache Spark > Flaky test: DataFrameRangeSuite >

[jira] [Assigned] (SPARK-23222) Flaky test: DataFrameRangeSuite

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23222: Assignee: (was: Apache Spark) > Flaky test: DataFrameRangeSuite >

[jira] [Commented] (SPARK-23222) Flaky test: DataFrameRangeSuite

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344548#comment-16344548 ] Apache Spark commented on SPARK-23222: -- User 'viirya' has created a pull request for this issue:

[jira] [Issue Comment Deleted] (SPARK-18016) Code Generation: Constant Pool Past Limit for Wide/Nested Dataset

2018-01-29 Thread Gaurav Garg (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gaurav Garg updated SPARK-18016: Comment: was deleted (was: [~kiszk], this programs also gives the Constant pool error in my

[jira] [Comment Edited] (SPARK-23252) When NodeManager and CoarseGrainedExecutorBackend processes are killed, the job will be blocked

2018-01-29 Thread Bang Xiao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344484#comment-16344484 ] Bang Xiao edited comment on SPARK-23252 at 1/30/18 4:30 AM: After the

[jira] [Commented] (SPARK-23252) When NodeManager and CoarseGrainedExecutorBackend processes are killed, the job will be blocked

2018-01-29 Thread Bang Xiao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344484#comment-16344484 ] Bang Xiao commented on SPARK-23252: --- After the executor and NodeManager is killed, failure tasks never

[jira] [Assigned] (SPARK-23263) create table stored as parquet should update table size if automatic update table size is enabled

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23263: Assignee: (was: Apache Spark) > create table stored as parquet should update table

[jira] [Commented] (SPARK-23263) create table stored as parquet should update table size if automatic update table size is enabled

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344460#comment-16344460 ] Apache Spark commented on SPARK-23263: -- User 'wangyum' has created a pull request for this issue:

[jira] [Assigned] (SPARK-23263) create table stored as parquet should update table size if automatic update table size is enabled

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23263: Assignee: Apache Spark > create table stored as parquet should update table size if

[jira] [Created] (SPARK-23263) create table stored as parquet should update table size if automatic update table size is enabled

2018-01-29 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-23263: --- Summary: create table stored as parquet should update table size if automatic update table size is enabled Key: SPARK-23263 URL: https://issues.apache.org/jira/browse/SPARK-23263

[jira] [Updated] (SPARK-23246) (Py)Spark OOM because of iteratively accumulated metadata that cannot be cleared

2018-01-29 Thread MBA Learns to Code (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] MBA Learns to Code updated SPARK-23246: --- Description: I am having consistent OOM crashes when trying to use PySpark for

[jira] [Resolved] (SPARK-23088) History server not showing incomplete/running applications

2018-01-29 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23088. - Resolution: Fixed Assignee: paul mackles Fix Version/s: 2.4.0 Issue resolved by

[jira] [Commented] (SPARK-23237) Add UI / endpoint for threaddumps for executors with active tasks

2018-01-29 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344434#comment-16344434 ] Imran Rashid commented on SPARK-23237: -- Can you expand a bit about what you are worried about?

[jira] [Comment Edited] (SPARK-23236) Make it easier to find the rest API, especially in local mode

2018-01-29 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344433#comment-16344433 ] Imran Rashid edited comment on SPARK-23236 at 1/30/18 3:09 AM: --- {quote} 1.

[jira] [Commented] (SPARK-23236) Make it easier to find the rest API, especially in local mode

2018-01-29 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344433#comment-16344433 ] Imran Rashid commented on SPARK-23236: -- bq. 1. /api and /api/v1 to give the same results (a

[jira] [Commented] (SPARK-23262) mix-in interface should extend the interface it aimed to mix in

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344396#comment-16344396 ] Apache Spark commented on SPARK-23262: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Assigned] (SPARK-23262) mix-in interface should extend the interface it aimed to mix in

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23262: Assignee: Wenchen Fan (was: Apache Spark) > mix-in interface should extend the interface

[jira] [Assigned] (SPARK-23262) mix-in interface should extend the interface it aimed to mix in

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23262: Assignee: Apache Spark (was: Wenchen Fan) > mix-in interface should extend the interface

[jira] [Created] (SPARK-23262) mix-in interface should extend the interface it aimed to mix in

2018-01-29 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-23262: --- Summary: mix-in interface should extend the interface it aimed to mix in Key: SPARK-23262 URL: https://issues.apache.org/jira/browse/SPARK-23262 Project: Spark

[jira] [Commented] (SPARK-18085) SPIP: Better History Server scalability for many / large applications

2018-01-29 Thread Alex Bozarth (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344339#comment-16344339 ] Alex Bozarth commented on SPARK-18085: -- [~vanzin] since this is complete and going into 2.3 I was

[jira] [Closed] (SPARK-21664) Use the column name as the file name.

2018-01-29 Thread jifei_yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jifei_yang closed SPARK-21664. -- We can use the partition to save the column names, such as: {code:java} case class

[jira] [Resolved] (SPARK-23246) (Py)Spark OOM because of iteratively accumulated metadata that cannot be cleared

2018-01-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-23246. --- Resolution: Not A Problem Yes, did you have a look? It's dominated by things like {{class

[jira] [Commented] (SPARK-23235) Add executor Threaddump to api

2018-01-29 Thread Alex Bozarth (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344168#comment-16344168 ] Alex Bozarth commented on SPARK-23235: -- Your discussion clarified my concern for me. I think I

[jira] [Commented] (SPARK-23235) Add executor Threaddump to api

2018-01-29 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344140#comment-16344140 ] Imran Rashid commented on SPARK-23235: -- [~ajbozarth] can you explain your concern? [~attilapiros]

[jira] [Assigned] (SPARK-23209) HiveDelegationTokenProvider throws an exception if Hive jars are not the classpath

2018-01-29 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-23209: Assignee: Marcelo Vanzin > HiveDelegationTokenProvider throws an exception if Hive jars

[jira] [Resolved] (SPARK-23209) HiveDelegationTokenProvider throws an exception if Hive jars are not the classpath

2018-01-29 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-23209. -- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 20399

[jira] [Assigned] (SPARK-23157) withColumn fails for a column that is a result of mapped DataSet

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23157: Assignee: Apache Spark > withColumn fails for a column that is a result of mapped DataSet

[jira] [Assigned] (SPARK-23157) withColumn fails for a column that is a result of mapped DataSet

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23157: Assignee: (was: Apache Spark) > withColumn fails for a column that is a result of

[jira] [Commented] (SPARK-23157) withColumn fails for a column that is a result of mapped DataSet

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344078#comment-16344078 ] Apache Spark commented on SPARK-23157: -- User 'henryr' has created a pull request for this issue:

[jira] [Assigned] (SPARK-23261) Rename Pandas UDFs

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23261: Assignee: Apache Spark (was: Xiao Li) > Rename Pandas UDFs > -- > >

[jira] [Assigned] (SPARK-23261) Rename Pandas UDFs

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23261: Assignee: Xiao Li (was: Apache Spark) > Rename Pandas UDFs > -- > >

[jira] [Commented] (SPARK-23261) Rename Pandas UDFs

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344076#comment-16344076 ] Apache Spark commented on SPARK-23261: -- User 'gatorsmile' has created a pull request for this issue:

[jira] [Comment Edited] (SPARK-23246) (Py)Spark OOM because of iteratively accumulated metadata that cannot be cleared

2018-01-29 Thread MBA Learns to Code (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344068#comment-16344068 ] MBA Learns to Code edited comment on SPARK-23246 at 1/29/18 9:45 PM: -

[jira] [Commented] (SPARK-23246) (Py)Spark OOM because of iteratively accumulated metadata that cannot be cleared

2018-01-29 Thread MBA Learns to Code (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344068#comment-16344068 ] MBA Learns to Code commented on SPARK-23246: [~srowen] the Java driver heap dump is attached

[jira] [Updated] (SPARK-23246) (Py)Spark OOM because of iteratively accumulated metadata that cannot be cleared

2018-01-29 Thread MBA Learns to Code (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] MBA Learns to Code updated SPARK-23246: --- Attachment: SparkProgramHeapDump.bin.tar.xz > (Py)Spark OOM because of iteratively

[jira] [Created] (SPARK-23261) Rename Pandas UDFs

2018-01-29 Thread Xiao Li (JIRA)
Xiao Li created SPARK-23261: --- Summary: Rename Pandas UDFs Key: SPARK-23261 URL: https://issues.apache.org/jira/browse/SPARK-23261 Project: Spark Issue Type: Sub-task Components: PySpark

[jira] [Commented] (SPARK-23157) withColumn fails for a column that is a result of mapped DataSet

2018-01-29 Thread Henry Robinson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343962#comment-16343962 ] Henry Robinson commented on SPARK-23157: [~kretes] - I can see an argument for the behaviour

[jira] [Commented] (SPARK-23260) remove V2 from the class name of data source reader/writer

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343878#comment-16343878 ] Apache Spark commented on SPARK-23260: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Assigned] (SPARK-23260) remove V2 from the class name of data source reader/writer

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23260: Assignee: Apache Spark (was: Wenchen Fan) > remove V2 from the class name of data source

[jira] [Assigned] (SPARK-23260) remove V2 from the class name of data source reader/writer

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23260: Assignee: Wenchen Fan (was: Apache Spark) > remove V2 from the class name of data source

[jira] [Created] (SPARK-23260) remove V2 from the class name of data source reader/writer

2018-01-29 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-23260: --- Summary: remove V2 from the class name of data source reader/writer Key: SPARK-23260 URL: https://issues.apache.org/jira/browse/SPARK-23260 Project: Spark

[jira] [Commented] (SPARK-23207) Shuffle+Repartition on an DataFrame could lead to incorrect answers

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343840#comment-16343840 ] Apache Spark commented on SPARK-23207: -- User 'jiangxb1987' has created a pull request for this

[jira] [Assigned] (SPARK-23259) Clean up legacy code around hive external catalog

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23259: Assignee: Apache Spark > Clean up legacy code around hive external catalog >

[jira] [Assigned] (SPARK-23259) Clean up legacy code around hive external catalog

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23259: Assignee: (was: Apache Spark) > Clean up legacy code around hive external catalog >

[jira] [Commented] (SPARK-23259) Clean up legacy code around hive external catalog

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343814#comment-16343814 ] Apache Spark commented on SPARK-23259: -- User 'liufengdb' has created a pull request for this issue:

[jira] [Created] (SPARK-23259) Clean up legacy code around hive external catalog

2018-01-29 Thread Feng Liu (JIRA)
Feng Liu created SPARK-23259: Summary: Clean up legacy code around hive external catalog Key: SPARK-23259 URL: https://issues.apache.org/jira/browse/SPARK-23259 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-23240) PythonWorkerFactory issues unhelpful message when pyspark.daemon produces bogus stdout

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23240: Assignee: (was: Apache Spark) > PythonWorkerFactory issues unhelpful message when

[jira] [Assigned] (SPARK-23240) PythonWorkerFactory issues unhelpful message when pyspark.daemon produces bogus stdout

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23240: Assignee: Apache Spark > PythonWorkerFactory issues unhelpful message when pyspark.daemon

[jira] [Commented] (SPARK-23240) PythonWorkerFactory issues unhelpful message when pyspark.daemon produces bogus stdout

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343792#comment-16343792 ] Apache Spark commented on SPARK-23240: -- User 'bersprockets' has created a pull request for this

[jira] [Commented] (SPARK-22221) Add User Documentation for Working with Arrow in Spark

2018-01-29 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343785#comment-16343785 ] Apache Spark commented on SPARK-1: -- User 'BryanCutler' has created a pull request for this

[jira] [Resolved] (SPARK-22221) Add User Documentation for Working with Arrow in Spark

2018-01-29 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-1. - Resolution: Fixed Assignee: Bryan Cutler Fix Version/s: 2.3.0 > Add User Documentation

[jira] [Created] (SPARK-23258) Should not split Arrow record batches based on row count

2018-01-29 Thread Bryan Cutler (JIRA)
Bryan Cutler created SPARK-23258: Summary: Should not split Arrow record batches based on row count Key: SPARK-23258 URL: https://issues.apache.org/jira/browse/SPARK-23258 Project: Spark

[jira] [Commented] (SPARK-23020) Re-enable Flaky Test: org.apache.spark.launcher.SparkLauncherSuite.testInProcessLauncher

2018-01-29 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343693#comment-16343693 ] Marcelo Vanzin commented on SPARK-23020: :-/ It's getting harder and harder to reproduce these

[jira] [Comment Edited] (SPARK-23109) ML 2.3 QA: API: Python API coverage

2018-01-29 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332698#comment-16332698 ] Bryan Cutler edited comment on SPARK-23109 at 1/29/18 5:26 PM: --- I did the

[jira] [Comment Edited] (SPARK-23109) ML 2.3 QA: API: Python API coverage

2018-01-29 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332698#comment-16332698 ] Bryan Cutler edited comment on SPARK-23109 at 1/29/18 5:25 PM: --- I did the

[jira] [Commented] (SPARK-23109) ML 2.3 QA: API: Python API coverage

2018-01-29 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343665#comment-16343665 ] Bryan Cutler commented on SPARK-23109: -- Thanks [~mlnick], yes this is done. > ML 2.3 QA: API:

[jira] [Resolved] (SPARK-23109) ML 2.3 QA: API: Python API coverage

2018-01-29 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler resolved SPARK-23109. -- Resolution: Done > ML 2.3 QA: API: Python API coverage > --- >

[jira] [Resolved] (SPARK-17006) WithColumn Performance Degrades with Number of Invocations

2018-01-29 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-17006. --- Resolution: Fixed Assignee: Herman van Hovell Fix Version/s: 2.3.0 >

[jira] [Resolved] (SPARK-23223) Stacking dataset transforms performs poorly

2018-01-29 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-23223. --- Resolution: Fixed Fix Version/s: 2.3.0 > Stacking dataset transforms performs

[jira] [Resolved] (SPARK-23059) Correct some improper with view related method usage

2018-01-29 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-23059. - Resolution: Fixed Fix Version/s: 2.4.0 > Correct some improper with view related method usage >

[jira] [Assigned] (SPARK-23059) Correct some improper with view related method usage

2018-01-29 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-23059: --- Assignee: xubo245 > Correct some improper with view related method usage >

[jira] [Resolved] (SPARK-23199) improved Removes repetition from group expressions in Aggregate

2018-01-29 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-23199. - Resolution: Fixed Assignee: caoxuewen Fix Version/s: 2.3.0 > improved Removes repetition

[jira] [Resolved] (SPARK-23219) Rename ReadTask to DataReaderFactory

2018-01-29 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-23219. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 20397

[jira] [Assigned] (SPARK-23219) Rename ReadTask to DataReaderFactory

2018-01-29 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-23219: --- Assignee: Gengliang Wang > Rename ReadTask to DataReaderFactory >

[jira] [Resolved] (SPARK-20129) JavaSparkContext should use SparkContext.getOrCreate

2018-01-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-20129. --- Resolution: Won't Fix Assignee: (was: Xiangrui Meng) Per PR discussion, I believe this

[jira] [Commented] (SPARK-23252) When NodeManager and CoarseGrainedExecutorBackend processes are killed, the job will be blocked

2018-01-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343358#comment-16343358 ] Sean Owen commented on SPARK-23252: --- That much looks normal if the executor is removed and the tasks

[jira] [Created] (SPARK-23257) Implement Kerberos Support in Kubernetes resource manager

2018-01-29 Thread Rob Keevil (JIRA)
Rob Keevil created SPARK-23257: -- Summary: Implement Kerberos Support in Kubernetes resource manager Key: SPARK-23257 URL: https://issues.apache.org/jira/browse/SPARK-23257 Project: Spark Issue

[jira] [Commented] (SPARK-23252) When NodeManager and CoarseGrainedExecutorBackend processes are killed, the job will be blocked

2018-01-29 Thread Bang Xiao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343317#comment-16343317 ] Bang Xiao commented on SPARK-23252: --- [~srowen] it seems the job  waits for the results of those tasks 

[jira] [Commented] (SPARK-23252) When NodeManager and CoarseGrainedExecutorBackend processes are killed, the job will be blocked

2018-01-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343288#comment-16343288 ] Sean Owen commented on SPARK-23252: --- Blocked how? Waiting for the NodeManager? YARN would know the NM

[jira] [Assigned] (SPARK-23108) ML, Graph 2.3 QA: API: Experimental, DeveloperApi, final, sealed audit

2018-01-29 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath reassigned SPARK-23108: -- Assignee: Nick Pentreath > ML, Graph 2.3 QA: API: Experimental, DeveloperApi, final,

[jira] [Comment Edited] (SPARK-23108) ML, Graph 2.3 QA: API: Experimental, DeveloperApi, final, sealed audit

2018-01-29 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343278#comment-16343278 ] Nick Pentreath edited comment on SPARK-23108 at 1/29/18 12:14 PM: -- Went

[jira] [Resolved] (SPARK-23108) ML, Graph 2.3 QA: API: Experimental, DeveloperApi, final, sealed audit

2018-01-29 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath resolved SPARK-23108. Resolution: Resolved Fix Version/s: 2.3.0 > ML, Graph 2.3 QA: API: Experimental,

[jira] [Commented] (SPARK-23108) ML, Graph 2.3 QA: API: Experimental, DeveloperApi, final, sealed audit

2018-01-29 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343290#comment-16343290 ] Nick Pentreath commented on SPARK-23108: Also checked ml {{DeveloperAPI}}, nothing to graduate

[jira] [Updated] (SPARK-23238) Externalize SQLConf spark.sql.execution.arrow.enabled

2018-01-29 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-23238: - Fix Version/s: 2.3.0 > Externalize SQLConf spark.sql.execution.arrow.enabled >

[jira] [Resolved] (SPARK-23238) Externalize SQLConf spark.sql.execution.arrow.enabled

2018-01-29 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-23238. -- Resolution: Fixed Fixed in https://github.com/apache/spark/pull/20403 > Externalize SQLConf

[jira] [Assigned] (SPARK-23238) Externalize SQLConf spark.sql.execution.arrow.enabled

2018-01-29 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-23238: Assignee: Hyukjin Kwon > Externalize SQLConf spark.sql.execution.arrow.enabled >

[jira] [Commented] (SPARK-23108) ML, Graph 2.3 QA: API: Experimental, DeveloperApi, final, sealed audit

2018-01-29 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343278#comment-16343278 ] Nick Pentreath commented on SPARK-23108: I think at this late stage we should not open up

[jira] [Commented] (SPARK-23157) withColumn fails for a column that is a result of mapped DataSet

2018-01-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343279#comment-16343279 ] Sean Owen commented on SPARK-23157: --- Agree this should not work . You are selecting a column from a

[jira] [Commented] (SPARK-23109) ML 2.3 QA: API: Python API coverage

2018-01-29 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343276#comment-16343276 ] Nick Pentreath commented on SPARK-23109: Created SPARK-23256 to track {{columnSchema}} in Python

[jira] [Created] (SPARK-23256) Add columnSchema method to PySpark image reader

2018-01-29 Thread Nick Pentreath (JIRA)
Nick Pentreath created SPARK-23256: -- Summary: Add columnSchema method to PySpark image reader Key: SPARK-23256 URL: https://issues.apache.org/jira/browse/SPARK-23256 Project: Spark Issue

[jira] [Commented] (SPARK-23109) ML 2.3 QA: API: Python API coverage

2018-01-29 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343269#comment-16343269 ] Nick Pentreath commented on SPARK-23109: So [~bryanc] I think this is done then? Can you confirm?

[jira] [Commented] (SPARK-21866) SPIP: Image support in Spark

2018-01-29 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343266#comment-16343266 ] Nick Pentreath commented on SPARK-21866: Ok, added SPARK-23255 to track user guide additions >

[jira] [Created] (SPARK-23255) Add user guide and examples for DataFrame image reading functions

2018-01-29 Thread Nick Pentreath (JIRA)
Nick Pentreath created SPARK-23255: -- Summary: Add user guide and examples for DataFrame image reading functions Key: SPARK-23255 URL: https://issues.apache.org/jira/browse/SPARK-23255 Project: Spark

[jira] [Updated] (SPARK-23107) ML, Graph 2.3 QA: API: New Scala APIs, docs

2018-01-29 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath updated SPARK-23107: --- Description: Audit new public Scala APIs added to MLlib & GraphX. Take note of: *

[jira] [Updated] (SPARK-23227) Add user guide entry for collecting sub models for cross-validation classes

2018-01-29 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath updated SPARK-23227: --- Priority: Minor (was: Major) > Add user guide entry for collecting sub models for

[jira] [Updated] (SPARK-23254) Add user guide entry for DataFrame multivariate summary

2018-01-29 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath updated SPARK-23254: --- Priority: Minor (was: Major) > Add user guide entry for DataFrame multivariate summary >

  1   2   >