[jira] [Commented] (SPARK-25258) Upgrade kryo package to version 4.0.2+

2018-08-27 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594557#comment-16594557 ] Yuming Wang commented on SPARK-25258: - I have submitted PR: 

[jira] [Commented] (SPARK-4502) Spark SQL reads unneccesary nested fields from Parquet

2018-08-27 Thread Damian Momot (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594546#comment-16594546 ] Damian Momot commented on SPARK-4502: - I can see that this ticket was closed, but by looking at

[jira] [Updated] (SPARK-25256) Plan mismatch errors in Hive tests in 2.12

2018-08-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-25256: - Shepherd: (was: Sean Owen) > Plan mismatch errors in Hive tests in 2.12 >

[jira] [Commented] (SPARK-25251) Make spark-csv's `quote` and `escape` options conform to RFC 4180

2018-08-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594522#comment-16594522 ] Hyukjin Kwon commented on SPARK-25251: -- This is a duplicate of SPARK-22236 > Make spark-csv's

[jira] [Resolved] (SPARK-25251) Make spark-csv's `quote` and `escape` options conform to RFC 4180

2018-08-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-25251. -- Resolution: Duplicate > Make spark-csv's `quote` and `escape` options conform to RFC 4180 >

[jira] [Commented] (SPARK-23255) Add user guide and examples for DataFrame image reading functions

2018-08-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594514#comment-16594514 ] Hyukjin Kwon commented on SPARK-23255: -- BTW, please be concise and precise on the documentation and

[jira] [Commented] (SPARK-23255) Add user guide and examples for DataFrame image reading functions

2018-08-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594513#comment-16594513 ] Hyukjin Kwon commented on SPARK-23255: -- Yea, I think so. For instance,

[jira] [Resolved] (SPARK-21232) New built-in SQL function - Data_Type

2018-08-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-21232. -- Resolution: Won't Fix Resolving this per the discussion in the JIRA > New built-in SQL

[jira] [Commented] (SPARK-23255) Add user guide and examples for DataFrame image reading functions

2018-08-27 Thread Divay Jindal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594488#comment-16594488 ] Divay Jindal commented on SPARK-23255: -- Hey, I have a very naive doubt, for this task do i need to

[jira] [Commented] (SPARK-24391) from_json should support arrays of primitives, and more generally all JSON

2018-08-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594479#comment-16594479 ] Hyukjin Kwon commented on SPARK-24391: -- For to_json, separate JIRA was filed in SPARK-25252 >

[jira] [Updated] (SPARK-24391) from_json should support arrays of primitives, and more generally all JSON

2018-08-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-24391: - Summary: from_json should support arrays of primitives, and more generally all JSON (was:

[jira] [Resolved] (SPARK-25213) DataSourceV2 doesn't seem to produce unsafe rows

2018-08-27 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-25213. - Resolution: Fixed Assignee: Li Jin Fix Version/s: 2.4.0 > DataSourceV2 doesn't

[jira] [Assigned] (SPARK-24721) Failed to use PythonUDF with literal inputs in filter with data sources

2018-08-27 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-24721: --- Assignee: Li Jin > Failed to use PythonUDF with literal inputs in filter with data sources

[jira] [Resolved] (SPARK-24721) Failed to use PythonUDF with literal inputs in filter with data sources

2018-08-27 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-24721. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22104

[jira] [Created] (SPARK-25258) Upgrade kryo package to version 4.0.2+

2018-08-27 Thread liupengcheng (JIRA)
liupengcheng created SPARK-25258: Summary: Upgrade kryo package to version 4.0.2+ Key: SPARK-25258 URL: https://issues.apache.org/jira/browse/SPARK-25258 Project: Spark Issue Type: Wish

[jira] [Updated] (SPARK-23207) Shuffle+Repartition on an DataFrame could lead to incorrect answers

2018-08-27 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23207: Fix Version/s: 2.1.4 > Shuffle+Repartition on an DataFrame could lead to incorrect answers >

[jira] [Updated] (SPARK-25164) Parquet reader builds entire list of columns once for each column

2018-08-27 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25164: Fix Version/s: 2.3.2 2.2.3 > Parquet reader builds entire list of columns once for

[jira] [Updated] (SPARK-25257) v2 MicroBatchReaders can't resume from checkpoints

2018-08-27 Thread Seth Fitzsimmons (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Fitzsimmons updated SPARK-25257: - Description: When resuming from a checkpoint: {code:java}

[jira] [Commented] (SPARK-25257) v2 MicroBatchReaders can't resume from checkpoints

2018-08-27 Thread Seth Fitzsimmons (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594298#comment-16594298 ] Seth Fitzsimmons commented on SPARK-25257: -- I traced this a bit further;

[jira] [Updated] (SPARK-25257) v2 MicroBatchReaders can't resume from checkpoints

2018-08-27 Thread Seth Fitzsimmons (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Fitzsimmons updated SPARK-25257: - Summary: v2 MicroBatchReaders can't resume from checkpoints (was:

[jira] [Updated] (SPARK-25257) java.lang.ClassCastException: org.apache.spark.sql.execution.streaming.SerializedOffset cannot be cast to org.apache.spark.sql.sources.v2.reader.streaming.Offset

2018-08-27 Thread Seth Fitzsimmons (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Fitzsimmons updated SPARK-25257: - Attachment: deserialize.patch > java.lang.ClassCastException: >

[jira] [Commented] (SPARK-25250) Race condition with tasks running when new attempt for same stage is created leads to other task in the next attempt running on the same partition id retry multiple ti

2018-08-27 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594281#comment-16594281 ] Thomas Graves commented on SPARK-25250: --- We are hitting a race condition here between the

[jira] [Commented] (SPARK-21097) Dynamic allocation will preserve cached data

2018-08-27 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594271#comment-16594271 ] Erik Erlandson commented on SPARK-21097: I'm wondering if this is going to be subsumed by the

[jira] [Created] (SPARK-25257) java.lang.ClassCastException: org.apache.spark.sql.execution.streaming.SerializedOffset cannot be cast to org.apache.spark.sql.sources.v2.reader.streaming.Offset

2018-08-27 Thread Seth Fitzsimmons (JIRA)
Seth Fitzsimmons created SPARK-25257: Summary: java.lang.ClassCastException: org.apache.spark.sql.execution.streaming.SerializedOffset cannot be cast to org.apache.spark.sql.sources.v2.reader.streaming.Offset Key:

[jira] [Resolved] (SPARK-24090) Kubernetes Backend Hotlist for Spark 2.4

2018-08-27 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-24090. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 4

[jira] [Commented] (SPARK-25175) Case-insensitive field resolution when reading from ORC

2018-08-27 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594213#comment-16594213 ] Dongjoon Hyun commented on SPARK-25175: --- Okay, thank you for the details, [~seancxmao]. BTW, for

[jira] [Created] (SPARK-25256) Plan mismatch errors in Hive tests in 2.12

2018-08-27 Thread Sean Owen (JIRA)
Sean Owen created SPARK-25256: - Summary: Plan mismatch errors in Hive tests in 2.12 Key: SPARK-25256 URL: https://issues.apache.org/jira/browse/SPARK-25256 Project: Spark Issue Type: Sub-task

[jira] [Assigned] (SPARK-25235) Merge the REPL code in Scala 2.11 and 2.12 branches

2018-08-27 Thread DB Tsai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DB Tsai reassigned SPARK-25235: --- Assignee: DB Tsai > Merge the REPL code in Scala 2.11 and 2.12 branches >

[jira] [Commented] (SPARK-16281) Implement parse_url SQL function

2018-08-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594169#comment-16594169 ] Apache Spark commented on SPARK-16281: -- User 'TomaszGaweda' has created a pull request for this

[jira] [Updated] (SPARK-25255) Add getActiveSession to SparkSession in PySpark

2018-08-27 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk updated SPARK-25255: Labels: starter (was: ) > Add getActiveSession to SparkSession in PySpark >

[jira] [Created] (SPARK-25255) Add getActiveSession to SparkSession in PySpark

2018-08-27 Thread holdenk (JIRA)
holdenk created SPARK-25255: --- Summary: Add getActiveSession to SparkSession in PySpark Key: SPARK-25255 URL: https://issues.apache.org/jira/browse/SPARK-25255 Project: Spark Issue Type:

[jira] [Created] (SPARK-25254) docker-image-tool should allow not building images for R and Python

2018-08-27 Thread Chaoran Yu (JIRA)
Chaoran Yu created SPARK-25254: -- Summary: docker-image-tool should allow not building images for R and Python Key: SPARK-25254 URL: https://issues.apache.org/jira/browse/SPARK-25254 Project: Spark

[jira] [Assigned] (SPARK-25253) Refactor pyspark connection & authentication

2018-08-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25253: Assignee: (was: Apache Spark) > Refactor pyspark connection & authentication >

[jira] [Assigned] (SPARK-25253) Refactor pyspark connection & authentication

2018-08-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25253: Assignee: Apache Spark > Refactor pyspark connection & authentication >

[jira] [Commented] (SPARK-25253) Refactor pyspark connection & authentication

2018-08-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594055#comment-16594055 ] Apache Spark commented on SPARK-25253: -- User 'squito' has created a pull request for this issue:

[jira] [Updated] (SPARK-25250) Race condition with tasks running when new attempt for same stage is created leads to other task in the next attempt running on the same partition id retry multiple time

2018-08-27 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-25250: -- Description: We recently had a scenario where a race condition occurred when a task from

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

2018-08-27 Thread Henry Robinson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594046#comment-16594046 ] Henry Robinson commented on SPARK-24434: Yeah, assignees are set after the PR is merged. I think

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

2018-08-27 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594045#comment-16594045 ] Stavros Kontopoulos commented on SPARK-24434: - Maybe it is added when the issue is resolved

[jira] [Updated] (SPARK-25250) Race condition with tasks running when new attempt for same stage is created leads to other task in the next attempt running on the same partition id retry multiple time

2018-08-27 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-25250: -- Priority: Major (was: Minor) > Race condition with tasks running when new attempt for same

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

2018-08-27 Thread Yinan Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594038#comment-16594038 ] Yinan Li commented on SPARK-24434: -- It seemed I couldn't change the assignee. > Support user-specified

[jira] [Commented] (SPARK-25235) Merge the REPL code in Scala 2.11 and 2.12 branches

2018-08-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594036#comment-16594036 ] Apache Spark commented on SPARK-25235: -- User 'dbtsai' has created a pull request for this issue:

[jira] [Assigned] (SPARK-25235) Merge the REPL code in Scala 2.11 and 2.12 branches

2018-08-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25235: Assignee: (was: Apache Spark) > Merge the REPL code in Scala 2.11 and 2.12 branches

[jira] [Assigned] (SPARK-25235) Merge the REPL code in Scala 2.11 and 2.12 branches

2018-08-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25235: Assignee: Apache Spark > Merge the REPL code in Scala 2.11 and 2.12 branches >

[jira] [Commented] (SPARK-24434) Support user-specified driver and executor pod templates

2018-08-27 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594024#comment-16594024 ] Stavros Kontopoulos commented on SPARK-24434: - Thanks [~liyinan926] I am reviewing the PR.

[jira] [Created] (SPARK-25253) Refactor pyspark connection & authentication

2018-08-27 Thread Imran Rashid (JIRA)
Imran Rashid created SPARK-25253: Summary: Refactor pyspark connection & authentication Key: SPARK-25253 URL: https://issues.apache.org/jira/browse/SPARK-25253 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-25252) Support arrays of any types in to_json

2018-08-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25252: Assignee: Apache Spark > Support arrays of any types in to_json >

[jira] [Assigned] (SPARK-25252) Support arrays of any types in to_json

2018-08-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25252: Assignee: (was: Apache Spark) > Support arrays of any types in to_json >

[jira] [Commented] (SPARK-25252) Support arrays of any types in to_json

2018-08-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594005#comment-16594005 ] Apache Spark commented on SPARK-25252: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Created] (SPARK-25252) Support arrays of any types in to_json

2018-08-27 Thread Maxim Gekk (JIRA)
Maxim Gekk created SPARK-25252: -- Summary: Support arrays of any types in to_json Key: SPARK-25252 URL: https://issues.apache.org/jira/browse/SPARK-25252 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-24882) data source v2 API improvement

2018-08-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593951#comment-16593951 ] Apache Spark commented on SPARK-24882: -- User 'jose-torres' has created a pull request for this

[jira] [Assigned] (SPARK-25249) Add a unit test for OpenHashMap

2018-08-27 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-25249: - Assignee: liuxian > Add a unit test for OpenHashMap > --- > >

[jira] [Resolved] (SPARK-25249) Add a unit test for OpenHashMap

2018-08-27 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-25249. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22241

[jira] [Updated] (SPARK-25249) Add a unit test for OpenHashMap

2018-08-27 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-25249: -- Priority: Trivial (was: Minor) I don't think this kind of thing needs a JIRA > Add a unit test for

[jira] [Updated] (SPARK-25240) A deadlock in ALTER TABLE RECOVER PARTITIONS

2018-08-27 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25240: Target Version/s: 2.4.0 > A deadlock in ALTER TABLE RECOVER PARTITIONS >

[jira] [Updated] (SPARK-25240) A deadlock in ALTER TABLE RECOVER PARTITIONS

2018-08-27 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25240: Priority: Blocker (was: Major) > A deadlock in ALTER TABLE RECOVER PARTITIONS >

[jira] [Created] (SPARK-25251) Make spark-csv's `quote` and `escape` options conform to RFC 4180

2018-08-27 Thread Ruslan Dautkhanov (JIRA)
Ruslan Dautkhanov created SPARK-25251: - Summary: Make spark-csv's `quote` and `escape` options conform to RFC 4180 Key: SPARK-25251 URL: https://issues.apache.org/jira/browse/SPARK-25251 Project:

[jira] [Commented] (SPARK-25091) UNCACHE TABLE, CLEAR CACHE, rdd.unpersist() does not clean up executor memory

2018-08-27 Thread Yunling Cai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593919#comment-16593919 ] Yunling Cai commented on SPARK-25091: - Thanks [~Chao Fang] for working on this! I have changed the

[jira] [Commented] (SPARK-23874) Upgrade apache/arrow to 0.10.0

2018-08-27 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593913#comment-16593913 ] Xiao Li commented on SPARK-23874: - ping [~bryanc] again. > Upgrade apache/arrow to 0.10.0 >

[jira] [Updated] (SPARK-25091) UNCACHE TABLE, CLEAR CACHE, rdd.unpersist() does not clean up executor memory

2018-08-27 Thread Yunling Cai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yunling Cai updated SPARK-25091: Summary: UNCACHE TABLE, CLEAR CACHE, rdd.unpersist() does not clean up executor memory (was:

[jira] [Commented] (SPARK-25091) Spark Thrift Server: UNCACHE TABLE and CLEAR CACHE does not clean up executor memory

2018-08-27 Thread Chao Fang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593908#comment-16593908 ] Chao Fang commented on SPARK-25091: --- hi [~dongjoon], Class AppStatusListener use LiveRDD and

[jira] [Comment Edited] (SPARK-25175) Case-insensitive field resolution when reading from ORC

2018-08-27 Thread Chenxiao Mao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593200#comment-16593200 ] Chenxiao Mao edited comment on SPARK-25175 at 8/27/18 3:53 PM: --- Also here

[jira] [Commented] (SPARK-25213) DataSourceV2 doesn't seem to produce unsafe rows

2018-08-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593858#comment-16593858 ] Apache Spark commented on SPARK-25213: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Commented] (SPARK-24721) Failed to use PythonUDF with literal inputs in filter with data sources

2018-08-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593857#comment-16593857 ] Apache Spark commented on SPARK-24721: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Updated] (SPARK-25206) wrong records are returned when Hive metastore schema and parquet schema are in different letter cases

2018-08-27 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25206: Target Version/s: 2.3.2, 2.4.0 (was: 2.3.2) > wrong records are returned when Hive metastore schema and

[jira] [Commented] (SPARK-25206) wrong records are returned when Hive metastore schema and parquet schema are in different letter cases

2018-08-27 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593840#comment-16593840 ] Xiao Li commented on SPARK-25206: - Silently ignoring it is bad. We should issue an exception like what

[jira] [Resolved] (SPARK-25227) Extend functionality of to_json to support arrays of differently-typed elements

2018-08-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-25227. -- Resolution: Duplicate > Extend functionality of to_json to support arrays of

[jira] [Commented] (SPARK-25226) Extend functionality of from_json to support arrays of differently-typed elements

2018-08-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593819#comment-16593819 ] Hyukjin Kwon commented on SPARK-25226: -- This is fixed in the current master: {code} >>> df =

[jira] [Resolved] (SPARK-25226) Extend functionality of from_json to support arrays of differently-typed elements

2018-08-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-25226. -- Resolution: Won't Fix > Extend functionality of from_json to support arrays of

[jira] [Commented] (SPARK-25226) Extend functionality of from_json to support arrays of differently-typed elements

2018-08-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593816#comment-16593816 ] Hyukjin Kwon commented on SPARK-25226: -- can you use: {code} >>> df = df.withColumn("parsed_data",

[jira] [Commented] (SPARK-25225) Add support for "List"-Type columns

2018-08-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593810#comment-16593810 ] Hyukjin Kwon commented on SPARK-25225: -- {quote} I coul cast it to string, but then later, when I

[jira] [Created] (SPARK-25250) Race condition with tasks running when new attempt for same stage is created leads to other task in the next attempt running on the same partition id retry multiple time

2018-08-27 Thread Parth Gandhi (JIRA)
Parth Gandhi created SPARK-25250: Summary: Race condition with tasks running when new attempt for same stage is created leads to other task in the next attempt running on the same partition id retry multiple times Key:

[jira] [Commented] (SPARK-24009) spark2.3.0 INSERT OVERWRITE LOCAL DIRECTORY '/home/spark/aaaaab'

2018-08-27 Thread Bang Xiao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593445#comment-16593445 ] Bang Xiao commented on SPARK-24009: --- any progress here?  i met the same error  sql: INSERT OVERWRITE

[jira] [Commented] (SPARK-25227) Extend functionality of to_json to support arrays of differently-typed elements

2018-08-27 Thread Maxim Gekk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593368#comment-16593368 ] Maxim Gekk commented on SPARK-25227: > I don't know about to_json. Maybe Maxim Gekk can comment more

[jira] [Commented] (SPARK-25227) Extend functionality of to_json to support arrays of differently-typed elements

2018-08-27 Thread Yuriy Davygora (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593286#comment-16593286 ] Yuriy Davygora commented on SPARK-25227: [~hyukjin.kwon] I only know, that in the upcoming

[jira] [Commented] (SPARK-25226) Extend functionality of from_json to support arrays of differently-typed elements

2018-08-27 Thread Yuriy Davygora (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593283#comment-16593283 ] Yuriy Davygora commented on SPARK-25226: [~hyukjin.kwon] Yes, sure. Here is some simple Python

[jira] [Commented] (SPARK-25225) Add support for "List"-Type columns

2018-08-27 Thread Yuriy Davygora (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593282#comment-16593282 ] Yuriy Davygora commented on SPARK-25225: [~maropu] Sorry, I was not quite clear on the ultimate

[jira] [Updated] (SPARK-25175) Case-insensitive field resolution when reading from ORC

2018-08-27 Thread Chenxiao Mao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chenxiao Mao updated SPARK-25175: - Description: SPARK-25132 adds support for case-insensitive field resolution when reading from

[jira] [Updated] (SPARK-25225) Add support for "List"-Type columns

2018-08-27 Thread Yuriy Davygora (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuriy Davygora updated SPARK-25225: --- Description: At the moment, Spark Dataframe ArrayType-columns only support all elements of

[jira] [Updated] (SPARK-25225) Add support for "List"-Type columns

2018-08-27 Thread Yuriy Davygora (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuriy Davygora updated SPARK-25225: --- Description: At the moment, Spark Dataframe ArrayType-columns only support all elements of

[jira] [Updated] (SPARK-25227) Extend functionality of to_json to support arrays of differently-typed elements

2018-08-27 Thread Yuriy Davygora (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuriy Davygora updated SPARK-25227: --- Summary: Extend functionality of to_json to support arrays of differently-typed elements

[jira] [Updated] (SPARK-25226) Extend functionality of from_json to support arrays of differently-typed elements

2018-08-27 Thread Yuriy Davygora (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuriy Davygora updated SPARK-25226: --- Summary: Extend functionality of from_json to support arrays of differently-typed elements

[jira] [Comment Edited] (SPARK-25175) Case-insensitive field resolution when reading from ORC

2018-08-27 Thread Chenxiao Mao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593194#comment-16593194 ] Chenxiao Mao edited comment on SPARK-25175 at 8/27/18 7:50 AM: ---

[jira] [Resolved] (SPARK-24978) Add spark.sql.fast.hash.aggregate.row.max.capacity to configure the capacity of fast aggregation.

2018-08-27 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-24978. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21931

[jira] [Assigned] (SPARK-24978) Add spark.sql.fast.hash.aggregate.row.max.capacity to configure the capacity of fast aggregation.

2018-08-27 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-24978: --- Assignee: caoxuewen > Add spark.sql.fast.hash.aggregate.row.max.capacity to configure the

[jira] [Commented] (SPARK-25249) Add a unit test for OpenHashMap

2018-08-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593268#comment-16593268 ] Apache Spark commented on SPARK-25249: -- User '10110346' has created a pull request for this issue:

[jira] [Assigned] (SPARK-25249) Add a unit test for OpenHashMap

2018-08-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25249: Assignee: (was: Apache Spark) > Add a unit test for OpenHashMap >

[jira] [Assigned] (SPARK-25249) Add a unit test for OpenHashMap

2018-08-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25249: Assignee: Apache Spark > Add a unit test for OpenHashMap >

[jira] [Comment Edited] (SPARK-25175) Case-insensitive field resolution when reading from ORC

2018-08-27 Thread Chenxiao Mao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593185#comment-16593185 ] Chenxiao Mao edited comment on SPARK-25175 at 8/27/18 7:33 AM: --- Thorough

[jira] [Created] (SPARK-25249) Add a unit test for OpenHashMap

2018-08-27 Thread liuxian (JIRA)
liuxian created SPARK-25249: --- Summary: Add a unit test for OpenHashMap Key: SPARK-25249 URL: https://issues.apache.org/jira/browse/SPARK-25249 Project: Spark Issue Type: Test Components:

[jira] [Comment Edited] (SPARK-25175) Case-insensitive field resolution when reading from ORC

2018-08-27 Thread Chenxiao Mao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593185#comment-16593185 ] Chenxiao Mao edited comment on SPARK-25175 at 8/27/18 7:24 AM: --- Thorough

[jira] [Updated] (SPARK-25175) Case-insensitive field resolution when reading from ORC

2018-08-27 Thread Chenxiao Mao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chenxiao Mao updated SPARK-25175: - Description: SPARK-25132 adds support for case-insensitive field resolution when reading from

[jira] [Updated] (SPARK-25175) Case-insensitive field resolution when reading from ORC

2018-08-27 Thread Chenxiao Mao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chenxiao Mao updated SPARK-25175: - Description: SPARK-25132 adds support for case-insensitive field resolution when reading from

[jira] [Comment Edited] (SPARK-25175) Case-insensitive field resolution when reading from ORC

2018-08-27 Thread Chenxiao Mao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593194#comment-16593194 ] Chenxiao Mao edited comment on SPARK-25175 at 8/27/18 6:56 AM: ---

[jira] [Comment Edited] (SPARK-25175) Case-insensitive field resolution when reading from ORC

2018-08-27 Thread Chenxiao Mao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593185#comment-16593185 ] Chenxiao Mao edited comment on SPARK-25175 at 8/27/18 6:45 AM: --- Thorough