[jira] [Comment Edited] (SPARK-23410) Unable to read jsons in charset different from UTF-8

2018-11-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700027#comment-16700027 ] Hyukjin Kwon edited comment on SPARK-23410 at 11/27/18 7:43 AM: I know

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

2018-11-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700027#comment-16700027 ] Hyukjin Kwon commented on SPARK-23410: -- I know BOM is only the beginning of the file .. just asked

[jira] [Commented] (SPARK-26164) [SQL] Allow FileFormatWriter to write multiple partitions/buckets without sort

2018-11-26 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700012#comment-16700012 ] Wenchen Fan commented on SPARK-26164: - The idea LGTM. It would be better to reuse some code between

[jira] [Commented] (SPARK-26164) [SQL] Allow FileFormatWriter to write multiple partitions/buckets without sort

2018-11-26 Thread Cheng Su (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1673#comment-1673 ] Cheng Su commented on SPARK-26164: -- _> I think we can follow aggregate, try hash writer first, and

[jira] [Commented] (SPARK-26181) the `hasMinMaxStats` method of `ColumnStatsMap` is not correct

2018-11-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699983#comment-16699983 ] Apache Spark commented on SPARK-26181: -- User 'adrian-wang' has created a pull request for this

[jira] [Resolved] (SPARK-26141) Enable custom shuffle metrics implementation in shuffle write

2018-11-26 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-26141. - Resolution: Fixed Fix Version/s: 3.0.0 > Enable custom shuffle metrics implementation in

[jira] [Assigned] (SPARK-26181) the `hasMinMaxStats` method of `ColumnStatsMap` is not correct

2018-11-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26181: Assignee: Apache Spark > the `hasMinMaxStats` method of `ColumnStatsMap` is not correct

[jira] [Commented] (SPARK-26181) the `hasMinMaxStats` method of `ColumnStatsMap` is not correct

2018-11-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699982#comment-16699982 ] Apache Spark commented on SPARK-26181: -- User 'adrian-wang' has created a pull request for this

[jira] [Assigned] (SPARK-26181) the `hasMinMaxStats` method of `ColumnStatsMap` is not correct

2018-11-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26181: Assignee: (was: Apache Spark) > the `hasMinMaxStats` method of `ColumnStatsMap` is

[jira] [Created] (SPARK-26181) the `hasMinMaxStats` method of `ColumnStatsMap` is not correct

2018-11-26 Thread Adrian Wang (JIRA)
Adrian Wang created SPARK-26181: --- Summary: the `hasMinMaxStats` method of `ColumnStatsMap` is not correct Key: SPARK-26181 URL: https://issues.apache.org/jira/browse/SPARK-26181 Project: Spark

[jira] [Commented] (SPARK-26155) Spark SQL performance degradation after apply SPARK-21052 with Q19 of TPC-DS in 3TB scale

2018-11-26 Thread Yang Jie (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699935#comment-16699935 ] Yang Jie commented on SPARK-26155: -- Is there any progress on this issue? > Spark SQL performance

[jira] [Assigned] (SPARK-26180) Add a withCreateTempDir function to the SparkCore test case

2018-11-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26180: Assignee: (was: Apache Spark) > Add a withCreateTempDir function to the SparkCore

[jira] [Commented] (SPARK-26180) Add a withCreateTempDir function to the SparkCore test case

2018-11-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699878#comment-16699878 ] Apache Spark commented on SPARK-26180: -- User 'heary-cao' has created a pull request for this issue:

[jira] [Assigned] (SPARK-26180) Add a withCreateTempDir function to the SparkCore test case

2018-11-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26180: Assignee: Apache Spark > Add a withCreateTempDir function to the SparkCore test case >

[jira] [Commented] (SPARK-26180) Add a withCreateTempDir function to the SparkCore test case

2018-11-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699879#comment-16699879 ] Apache Spark commented on SPARK-26180: -- User 'heary-cao' has created a pull request for this issue:

[jira] [Created] (SPARK-26180) Add a withCreateTempDir function to the SparkCore test case

2018-11-26 Thread caoxuewen (JIRA)
caoxuewen created SPARK-26180: - Summary: Add a withCreateTempDir function to the SparkCore test case Key: SPARK-26180 URL: https://issues.apache.org/jira/browse/SPARK-26180 Project: Spark Issue

[jira] [Comment Edited] (SPARK-23410) Unable to read jsons in charset different from UTF-8

2018-11-26 Thread xuqianjin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699803#comment-16699803 ] xuqianjin edited comment on SPARK-23410 at 11/27/18 1:59 AM: - hi [~maxgekk]

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

2018-11-26 Thread xuqianjin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699803#comment-16699803 ] xuqianjin commented on SPARK-23410: --- hi [~maxgekk] [~hyukjin.kwon] I think there are two things to

[jira] [Commented] (SPARK-19256) Hive bucketing support

2018-11-26 Thread Cheng Su (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699763#comment-16699763 ] Cheng Su commented on SPARK-19256: -- After discussion with [~tejasp], I think the most of un-merged pr

[jira] [Updated] (SPARK-26164) [SQL] Allow FileFormatWriter to write multiple partitions/buckets without sort

2018-11-26 Thread Cheng Su (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Su updated SPARK-26164: - Issue Type: Sub-task (was: Improvement) Parent: SPARK-19256 > [SQL] Allow FileFormatWriter to

[jira] [Updated] (SPARK-26179) `map_concat` should replace the value in the left side

2018-11-26 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-26179: Description: Spark SQL map is internally represented by two arrays, and when two maps are concatenated,

[jira] [Updated] (SPARK-26179) `map_concat` should replace the value in the left side

2018-11-26 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-26179: Description: Spark SQL map is internally represented by two arrays, and when two maps are concatenated,

[jira] [Updated] (SPARK-26179) `map_concat` should replace the value in the left side

2018-11-26 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-26179: Description: See the following example,   {noformat} import org.apache.spark.sql.functions.udf val

[jira] [Created] (SPARK-26179) `map_concat` should replace the value in the left side

2018-11-26 Thread DB Tsai (JIRA)
DB Tsai created SPARK-26179: --- Summary: `map_concat` should replace the value in the left side Key: SPARK-26179 URL: https://issues.apache.org/jira/browse/SPARK-26179 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-26179) `map_concat` should replace the value in the left side

2018-11-26 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai updated SPARK-26179: Description: See the following example,   {code:scala} import org.apache.spark.sql.functions.udf val

[jira] [Commented] (SPARK-26178) Use java.time API for parsing timestamps and dates from CSV

2018-11-26 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699718#comment-16699718 ] Sean Owen commented on SPARK-26178: --- Does this also resolve SPARK-19228? The original PR was tagged

[jira] [Assigned] (SPARK-26178) Use java.time API for parsing timestamps and dates from CSV

2018-11-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26178: Assignee: Apache Spark > Use java.time API for parsing timestamps and dates from CSV >

[jira] [Assigned] (SPARK-26178) Use java.time API for parsing timestamps and dates from CSV

2018-11-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26178: Assignee: (was: Apache Spark) > Use java.time API for parsing timestamps and dates

[jira] [Commented] (SPARK-26178) Use java.time API for parsing timestamps and dates from CSV

2018-11-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699678#comment-16699678 ] Apache Spark commented on SPARK-26178: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Commented] (SPARK-26089) Handle large corrupt shuffle blocks

2018-11-26 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699670#comment-16699670 ] Imran Rashid commented on SPARK-26089: -- yeah the stacktrace is basically the same as SPARK-4105.

[jira] [Created] (SPARK-26178) Use java.time API for parsing timestamps and dates from CSV

2018-11-26 Thread Maxim Gekk (JIRA)
Maxim Gekk created SPARK-26178: -- Summary: Use java.time API for parsing timestamps and dates from CSV Key: SPARK-26178 URL: https://issues.apache.org/jira/browse/SPARK-26178 Project: Spark

[jira] [Commented] (SPARK-26089) Handle large corrupt shuffle blocks

2018-11-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699639#comment-16699639 ] Thomas Graves commented on SPARK-26089: --- it would definitely be nice to improve blacklisting,

[jira] [Commented] (SPARK-25451) Stages page doesn't show the right number of the total tasks

2018-11-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699638#comment-16699638 ] Apache Spark commented on SPARK-25451: -- User 'vanzin' has created a pull request for this issue:

[jira] [Resolved] (SPARK-25451) Stages page doesn't show the right number of the total tasks

2018-11-26 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25451. Resolution: Fixed Assignee: shahid Fix Version/s: 3.0.0

[jira] [Resolved] (SPARK-26100) [History server ]Jobs table and Aggregate metrics table are showing lesser number of tasks

2018-11-26 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26100. Resolution: Fixed Assignee: shahid Fix Version/s: 3.0.0

[jira] [Assigned] (SPARK-26177) Automated formatting for Scala code

2018-11-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26177: Assignee: (was: Apache Spark) > Automated formatting for Scala code >

[jira] [Commented] (SPARK-26177) Automated formatting for Scala code

2018-11-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699536#comment-16699536 ] Apache Spark commented on SPARK-26177: -- User 'koeninger' has created a pull request for this issue:

[jira] [Commented] (SPARK-26177) Automated formatting for Scala code

2018-11-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699534#comment-16699534 ] Apache Spark commented on SPARK-26177: -- User 'koeninger' has created a pull request for this issue:

[jira] [Assigned] (SPARK-26177) Automated formatting for Scala code

2018-11-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26177: Assignee: Apache Spark > Automated formatting for Scala code >

[jira] [Resolved] (SPARK-21809) Change Stage Page to use datatables to support sorting columns and searching

2018-11-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-21809. --- Resolution: Fixed Assignee: Parth Gandhi Fix Version/s: 3.0.0 > Change

[jira] [Created] (SPARK-26177) Automated formatting for Scala code

2018-11-26 Thread Cody Koeninger (JIRA)
Cody Koeninger created SPARK-26177: -- Summary: Automated formatting for Scala code Key: SPARK-26177 URL: https://issues.apache.org/jira/browse/SPARK-26177 Project: Spark Issue Type:

[jira] [Commented] (SPARK-26140) Enable custom shuffle metrics implementation in shuffle reader

2018-11-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699444#comment-16699444 ] Apache Spark commented on SPARK-26140: -- User 'rxin' has created a pull request for this issue:

[jira] [Resolved] (SPARK-25960) Support subpath mounting with Kubernetes

2018-11-26 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-25960. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23026

[jira] [Assigned] (SPARK-25960) Support subpath mounting with Kubernetes

2018-11-26 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-25960: -- Assignee: Nihar Sheth > Support subpath mounting with Kubernetes >

[jira] [Commented] (SPARK-26176) Verify column name when creating table via `STORED AS`

2018-11-26 Thread kevin yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699370#comment-16699370 ] kevin yu commented on SPARK-26176: -- I will look into it. Kevin > Verify column name when creating

[jira] [Commented] (SPARK-26175) PySpark cannot terminate worker process if user program reads from stdin

2018-11-26 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699339#comment-16699339 ] Xiangrui Meng commented on SPARK-26175: --- This affects Hydrogen because the external training

[jira] [Updated] (SPARK-26175) PySpark cannot terminate worker process if user program reads from stdin

2018-11-26 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-26175: -- Labels: Hydrogen (was: ) > PySpark cannot terminate worker process if user program reads

[jira] [Commented] (SPARK-26175) PySpark cannot terminate worker process if user program reads from stdin

2018-11-26 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699337#comment-16699337 ] Xiao Li commented on SPARK-26175: - cc [~hyukjin.kwon] [~bryanc] [~icexelloss] > PySpark cannot

[jira] [Updated] (SPARK-26175) PySpark cannot terminate worker process if user program reads from stdin

2018-11-26 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-26175: Target Version/s: 3.0.0 > PySpark cannot terminate worker process if user program reads from stdin >

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

2018-11-26 Thread Maxim Gekk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699333#comment-16699333 ] Maxim Gekk commented on SPARK-23410: > Every line has the BOM? BOM can be only at the beginning of

[jira] [Updated] (SPARK-26152) Flaky test: BroadcastSuite

2018-11-26 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26152: -- Priority: Critical (was: Blocker) > Flaky test: BroadcastSuite > --

[jira] [Updated] (SPARK-26176) Verify column name when creating table via `STORED AS`

2018-11-26 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-26176: Issue Type: Bug (was: Test) > Verify column name when creating table via `STORED AS` >

[jira] [Updated] (SPARK-26176) Verify column name when creating table via `STORED AS`

2018-11-26 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-26176: Labels: starter (was: ) > Verify column name when creating table via `STORED AS` >

[jira] [Created] (SPARK-26176) Verify column name when creating table via `STORED AS`

2018-11-26 Thread Xiao Li (JIRA)
Xiao Li created SPARK-26176: --- Summary: Verify column name when creating table via `STORED AS` Key: SPARK-26176 URL: https://issues.apache.org/jira/browse/SPARK-26176 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-26121) [Structured Streaming] Allow users to define prefix of Kafka's consumer group (group.id)

2018-11-26 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cody Koeninger resolved SPARK-26121. Resolution: Fixed Assignee: Anastasios Zouzias Fix Version/s: 3.0.0

[jira] [Created] (SPARK-26175) PySpark cannot terminate worker process if user program reads from stdin

2018-11-26 Thread Ala Luszczak (JIRA)
Ala Luszczak created SPARK-26175: Summary: PySpark cannot terminate worker process if user program reads from stdin Key: SPARK-26175 URL: https://issues.apache.org/jira/browse/SPARK-26175 Project:

[jira] [Commented] (SPARK-26143) Shuffle shuffle default storage level

2018-11-26 Thread Avi minsky (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699294#comment-16699294 ] Avi minsky commented on SPARK-26143: Thank you, I will > Shuffle shuffle default storage level >

[jira] [Commented] (SPARK-26143) Shuffle shuffle default storage level

2018-11-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699293#comment-16699293 ] Hyukjin Kwon commented on SPARK-26143: -- Feature requests are a-okay but these are generally not

[jira] [Commented] (SPARK-26143) Shuffle shuffle default storage level

2018-11-26 Thread Avi minsky (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699292#comment-16699292 ] Avi minsky commented on SPARK-26143: I'm not fully familiar with spark internals, I'm a spark user

[jira] [Commented] (SPARK-26143) Shuffle shuffle default storage level

2018-11-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699278#comment-16699278 ] Hyukjin Kwon commented on SPARK-26143: -- Right, are you willing to investigate and share if that's

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

2018-11-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699272#comment-16699272 ] Hyukjin Kwon commented on SPARK-23410: -- There look no discussion made about it in that project. I

[jira] [Commented] (SPARK-26143) Shuffle shuffle default storage level

2018-11-26 Thread Avi minsky (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699267#comment-16699267 ] Avi minsky commented on SPARK-26143: It's a feature request more than anything. The main issue I'm

[jira] [Commented] (SPARK-26171) Is Spark 2.3+ support UDT?

2018-11-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699261#comment-16699261 ] Hyukjin Kwon commented on SPARK-26171: -- This link will be helpful -

[jira] [Commented] (SPARK-26171) Is Spark 2.3+ support UDT?

2018-11-26 Thread SUNY TYAGI (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699258#comment-16699258 ] SUNY TYAGI commented on SPARK-26171: [~hyukjin.kwon]  Thanks for quick response.Could you please

[jira] [Commented] (SPARK-26173) Prior regularization for Logistic Regression

2018-11-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699237#comment-16699237 ] Apache Spark commented on SPARK-26173: -- User 'elfausto' has created a pull request for this issue:

[jira] [Commented] (SPARK-26160) Make assertNotBucketed call in DataFrameWriter::save optional

2018-11-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699255#comment-16699255 ] Hyukjin Kwon commented on SPARK-26160: -- Let's ask it / start the discussion from mailing list. >

[jira] [Resolved] (SPARK-26157) Asynchronous execution of stored procedure

2018-11-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-26157. -- Resolution: Invalid Looks a question. Questions should go to mailing list. I think you could

[jira] [Commented] (SPARK-26155) Spark SQL performance degradation after apply SPARK-21052 with Q19 of TPC-DS in 3TB scale

2018-11-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699248#comment-16699248 ] Hyukjin Kwon commented on SPARK-26155: -- cc [~viirya] FYI > Spark SQL performance degradation

[jira] [Commented] (SPARK-26152) Flaky test: BroadcastSuite

2018-11-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699246#comment-16699246 ] Hyukjin Kwon commented on SPARK-26152: -- [~dongjoon], btw, why is this flaky test a release blocker?

[jira] [Commented] (SPARK-26167) No output created for aggregation query in append mode

2018-11-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699228#comment-16699228 ] Hyukjin Kwon commented on SPARK-26167: -- Please avoid to set Critical+ priority which is usually

[jira] [Commented] (SPARK-26171) Is Spark 2.3+ support UDT?

2018-11-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699226#comment-16699226 ] Hyukjin Kwon commented on SPARK-26171: -- Questions should better go to mailing list. Let's only file

[jira] [Commented] (SPARK-26103) OutOfMemory error with large query plans

2018-11-26 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699239#comment-16699239 ] Marcelo Vanzin commented on SPARK-26103: The proposed patch would probably also fix SPARK-25380.

[jira] [Commented] (SPARK-26146) CSV wouln't be ingested in Spark 2.4.0 with Scala 2.12

2018-11-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699238#comment-16699238 ] Hyukjin Kwon commented on SPARK-26146: -- Would you mind if I ask to make a reproducer with

[jira] [Assigned] (SPARK-26173) Prior regularization for Logistic Regression

2018-11-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26173: Assignee: (was: Apache Spark) > Prior regularization for Logistic Regression >

[jira] [Commented] (SPARK-26173) Prior regularization for Logistic Regression

2018-11-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699235#comment-16699235 ] Apache Spark commented on SPARK-26173: -- User 'elfausto' has created a pull request for this issue:

[jira] [Assigned] (SPARK-26173) Prior regularization for Logistic Regression

2018-11-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26173: Assignee: Apache Spark > Prior regularization for Logistic Regression >

[jira] [Commented] (SPARK-26143) Shuffle shuffle default storage level

2018-11-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699229#comment-16699229 ] Hyukjin Kwon commented on SPARK-26143: -- Is it a question or an issue? > Shuffle shuffle default

[jira] [Resolved] (SPARK-26145) Not Able To Read Data From Hive 3.0 Using Spark 2.3

2018-11-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-26145. -- Resolution: Duplicate > Not Able To Read Data From Hive 3.0 Using Spark 2.3 >

[jira] [Updated] (SPARK-26167) No output created for aggregation query in append mode

2018-11-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-26167: - Priority: Major (was: Blocker) > No output created for aggregation query in append mode >

[jira] [Resolved] (SPARK-26167) No output created for aggregation query in append mode

2018-11-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-26167. -- Resolution: Cannot Reproduce > No output created for aggregation query in append mode >

[jira] [Resolved] (SPARK-26171) Is Spark 2.3+ support UDT?

2018-11-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-26171. -- Resolution: Invalid > Is Spark 2.3+ support UDT? > -- > >

[jira] [Created] (SPARK-26174) fail when reading parquet map column with duplicated keys

2018-11-26 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-26174: --- Summary: fail when reading parquet map column with duplicated keys Key: SPARK-26174 URL: https://issues.apache.org/jira/browse/SPARK-26174 Project: Spark

[jira] [Updated] (SPARK-24434) Support user-specified driver and executor pod templates

2018-11-26 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson updated SPARK-24434: --- Fix Version/s: 3.0.0 > Support user-specified driver and executor pod templates >

[jira] [Commented] (SPARK-18180) pyspark.sql.Row does not serialize well to json

2018-11-26 Thread Oleg V Korchagin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698965#comment-16698965 ] Oleg V Korchagin commented on SPARK-18180: -- {code:java} In [1]: from pyspark.sql.types import

[jira] [Created] (SPARK-26173) Prior regularization for Logistic Regression

2018-11-26 Thread Facundo Bellosi (JIRA)
Facundo Bellosi created SPARK-26173: --- Summary: Prior regularization for Logistic Regression Key: SPARK-26173 URL: https://issues.apache.org/jira/browse/SPARK-26173 Project: Spark Issue

[jira] [Resolved] (SPARK-25729) It is better to replace `minPartitions` with `defaultParallelism` , when `minPartitions` is less than `defaultParallelism`

2018-11-26 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-25729. --- Resolution: Won't Fix > It is better to replace `minPartitions` with `defaultParallelism` , when >

[jira] [Commented] (SPARK-25590) kubernetes-model-2.0.0.jar masks default Spark logging config

2018-11-26 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698866#comment-16698866 ] Stavros Kontopoulos commented on SPARK-25590: - As mentioned elsewhere I think: --conf

[jira] [Resolved] (SPARK-26153) GBT & RandomForest avoid unnecessary `first` job to compute `numFeatures`

2018-11-26 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-26153. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23123

[jira] [Assigned] (SPARK-26153) GBT & RandomForest avoid unnecessary `first` job to compute `numFeatures`

2018-11-26 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-26153: - Assignee: zhengruifeng > GBT & RandomForest avoid unnecessary `first` job to compute

[jira] [Commented] (SPARK-26172) Unify String Params' case-insensitivity in ML

2018-11-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698707#comment-16698707 ] Apache Spark commented on SPARK-26172: -- User 'zhengruifeng' has created a pull request for this

[jira] [Updated] (SPARK-26171) Is Spark 2.3+ support UDT?

2018-11-26 Thread SUNY TYAGI (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SUNY TYAGI updated SPARK-26171: --- Description: I was going through this ticket

[jira] [Updated] (SPARK-26172) Unify String Params' case-insensitivity in ML

2018-11-26 Thread zhengruifeng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng updated SPARK-26172: - Description: For now, there are three ways to deal with case-insensitivity in ML: 1, support 

[jira] [Commented] (SPARK-26172) Unify String Params' case-insensitivity in ML

2018-11-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698706#comment-16698706 ] Apache Spark commented on SPARK-26172: -- User 'zhengruifeng' has created a pull request for this

[jira] [Assigned] (SPARK-26172) Unify String Params' case-insensitivity in ML

2018-11-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26172: Assignee: (was: Apache Spark) > Unify String Params' case-insensitivity in ML >

[jira] [Assigned] (SPARK-26172) Unify String Params' case-insensitivity in ML

2018-11-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26172: Assignee: Apache Spark > Unify String Params' case-insensitivity in ML >

[jira] [Updated] (SPARK-26172) Unify String Params' case-insensitivity in ML

2018-11-26 Thread zhengruifeng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng updated SPARK-26172: - Description: For now, there are three ways to deal with case-insensitivity in ML: 1, support 

[jira] [Updated] (SPARK-26172) Unify String Params' case-insensitivity in ML

2018-11-26 Thread zhengruifeng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng updated SPARK-26172: - Description: For now, there are three ways to deal with case-insensitivity in ML: 1, support 

[jira] [Updated] (SPARK-26171) Is Spark 2.3+ support UDT?

2018-11-26 Thread SUNY TYAGI (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SUNY TYAGI updated SPARK-26171: --- Summary: Is Spark 2.3+ support UDT? (was: Is Spark support UDT?) > Is Spark 2.3+ support UDT? >

[jira] [Created] (SPARK-26172) Unify String Params' case-insensitivity in ML

2018-11-26 Thread zhengruifeng (JIRA)
zhengruifeng created SPARK-26172: Summary: Unify String Params' case-insensitivity in ML Key: SPARK-26172 URL: https://issues.apache.org/jira/browse/SPARK-26172 Project: Spark Issue Type:

[jira] [Updated] (SPARK-26172) Unify String Params' case-insensitivity in ML

2018-11-26 Thread zhengruifeng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng updated SPARK-26172: - Description: For now, there are three ways to deal with case-insensitivity in ML: 1, support 

  1   2   >