[jira] [Assigned] (SPARK-24886) Increase Jenkins build time

2018-07-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24886: Assignee: Apache Spark > Increase Jenkins build time > --- > >

[jira] [Commented] (SPARK-24886) Increase Jenkins build time

2018-07-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16552347#comment-16552347 ] Apache Spark commented on SPARK-24886: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-24886) Increase Jenkins build time

2018-07-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24886: Assignee: (was: Apache Spark) > Increase Jenkins build time >

[jira] [Created] (SPARK-24886) Increase Jenkins build time

2018-07-22 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-24886: Summary: Increase Jenkins build time Key: SPARK-24886 URL: https://issues.apache.org/jira/browse/SPARK-24886 Project: Spark Issue Type: Test

[jira] [Commented] (SPARK-18492) GeneratedIterator grows beyond 64 KB

2018-07-22 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16552257#comment-16552257 ] Takeshi Yamamuro commented on SPARK-18492: -- [~MDS Tang] You'd be better to describe more about

[jira] [Commented] (SPARK-18492) GeneratedIterator grows beyond 64 KB

2018-07-22 Thread Tang Yu Jie (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16552242#comment-16552242 ] Tang Yu Jie commented on SPARK-18492: - Here I also encountered this problem, have you resolved it?  

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

2018-07-22 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16552233#comment-16552233 ] Saisai Shao commented on SPARK-24615: - Hi [~tgraves] what you mentioned above is also what we think

[jira] [Commented] (SPARK-24841) Memory leak in converting spark dataframe to pandas dataframe

2018-07-22 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16552231#comment-16552231 ] Kazuaki Ishizaki commented on SPARK-24841: -- Thank you for reporting an issue with heap

[jira] [Resolved] (SPARK-24859) Predicates pushdown on outer joins

2018-07-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24859. -- Resolution: Cannot Reproduce Seems fixed in the master branch. Please let me know if you have

[jira] [Resolved] (SPARK-24847) ScalaReflection#schemaFor occasionally fails to detect schema for Seq of type alias

2018-07-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24847. -- Resolution: Cannot Reproduce > ScalaReflection#schemaFor occasionally fails to detect schema

[jira] [Resolved] (SPARK-24853) Support Column type for withColumn and withColumnRenamed apis

2018-07-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24853. -- Resolution: Won't Fix > Support Column type for withColumn and withColumnRenamed apis >

[jira] [Updated] (SPARK-24884) Implement regexp_extract_all

2018-07-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-24884: - Component/s: (was: Spark Core) SQL > Implement regexp_extract_all >

[jira] [Resolved] (SPARK-24885) Initialize random seeds for Rand and Randn expression during analysis

2018-07-22 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh resolved SPARK-24885. - Resolution: Won't Fix > Initialize random seeds for Rand and Randn expression during

[jira] [Updated] (SPARK-24885) Initialize random seeds for Rand and Randn expression during analysis

2018-07-22 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-24885: Description: Random expressions such as Rand and Randn should have the same behavior as

[jira] [Updated] (SPARK-24885) Initialize random seeds for Rand and Randn expression during analysis

2018-07-22 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-24885: Summary: Initialize random seeds for Rand and Randn expression during analysis (was:

[jira] [Updated] (SPARK-24885) Rand and Randn expression should produce same result at DataFrame on retries

2018-07-22 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-24885: Description: Random expressions such as Rand and Randn should have the same behavior as

[jira] [Created] (SPARK-24885) Rand and Randn expression should produce same result at DataFrame on retries

2018-07-22 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-24885: --- Summary: Rand and Randn expression should produce same result at DataFrame on retries Key: SPARK-24885 URL: https://issues.apache.org/jira/browse/SPARK-24885

[jira] [Resolved] (SPARK-22228) Add support for Array so from_json can parse

2018-07-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-8. -- Resolution: Duplicate > Add support for Array so from_json can parse >

[jira] [Commented] (SPARK-16203) regexp_extract to return an ArrayType(StringType())

2018-07-22 Thread Nick Nicolini (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16552155#comment-16552155 ] Nick Nicolini commented on SPARK-16203: --- Cool, added ticket

[jira] [Updated] (SPARK-24884) Implement regexp_extract_all

2018-07-22 Thread Nick Nicolini (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Nicolini updated SPARK-24884: -- Description: I've recently hit many cases of regexp parsing where we need to match on

[jira] [Created] (SPARK-24884) Implement regexp_extract_all

2018-07-22 Thread Nick Nicolini (JIRA)
Nick Nicolini created SPARK-24884: - Summary: Implement regexp_extract_all Key: SPARK-24884 URL: https://issues.apache.org/jira/browse/SPARK-24884 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-24884) Implement regexp_extract_all

2018-07-22 Thread Nick Nicolini (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Nicolini updated SPARK-24884: -- Description: I've recently hit many cases of regexp parsing where we need to match on

[jira] [Updated] (SPARK-16483) Unifying struct fields and columns

2018-07-22 Thread Simeon Simeonov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simeon Simeonov updated SPARK-16483: Description: This issue comes as a result of an exchange with Michael Armbrust outside of

[jira] [Commented] (SPARK-24768) Have a built-in AVRO data source implementation

2018-07-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16552125#comment-16552125 ] Apache Spark commented on SPARK-24768: -- User 'gengliangwang' has created a pull request for this

[jira] [Created] (SPARK-24883) Remove implicit class AvroDataFrameWriter/AvroDataFrameReader

2018-07-22 Thread Gengliang Wang (JIRA)
Gengliang Wang created SPARK-24883: -- Summary: Remove implicit class AvroDataFrameWriter/AvroDataFrameReader Key: SPARK-24883 URL: https://issues.apache.org/jira/browse/SPARK-24883 Project: Spark

[jira] [Commented] (SPARK-24869) SaveIntoDataSourceCommand's input Dataset does not use Cached Data

2018-07-22 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16552103#comment-16552103 ] Xiao Li commented on SPARK-24869: - [~maropu] Just updated the test case in the JIRA. Forgot to cache it

[jira] [Updated] (SPARK-24869) SaveIntoDataSourceCommand's input Dataset does not use Cached Data

2018-07-22 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24869: Description: {code} withTable("t") { withTempPath { path => var numTotalCachedHit = 0

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

2018-07-22 Thread James (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16552087#comment-16552087 ] James commented on SPARK-23206: --- Hi  [~elu] If I want to know the CPU metrics of executor level, what

[jira] [Updated] (SPARK-16483) Unifying struct fields and columns

2018-07-22 Thread Simeon Simeonov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simeon Simeonov updated SPARK-16483: Affects Version/s: 2.3.1 Description: This issue comes as a result of an

[jira] [Resolved] (SPARK-22562) CachedKafkaConsumer unsafe eviction from cache

2018-07-22 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-22562. --- Resolution: Not A Problem > CachedKafkaConsumer unsafe eviction from cache >

[jira] [Commented] (SPARK-24882) separate responsibilities of the data source v2 read API

2018-07-22 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16552041#comment-16552041 ] Wenchen Fan commented on SPARK-24882: - cc people we may be interested: [~rxin] [~rdblue] [~marmbrus]

[jira] [Comment Edited] (SPARK-24882) separate responsibilities of the data source v2 read API

2018-07-22 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16552041#comment-16552041 ] Wenchen Fan edited comment on SPARK-24882 at 7/22/18 2:49 PM: -- cc people

[jira] [Commented] (SPARK-24339) spark sql can not prune column in transform/map/reduce query

2018-07-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16552040#comment-16552040 ] Apache Spark commented on SPARK-24339: -- User 'xuanyuanking' has created a pull request for this

[jira] [Updated] (SPARK-24882) separate responsibilities of the data source v2 read API

2018-07-22 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-24882: External issue URL: (was:

[jira] [Updated] (SPARK-24882) separate responsibilities of the data source v2 read API

2018-07-22 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-24882: Description: Data source V2 is out for a while, see the SPIP

[jira] [Updated] (SPARK-24882) separate responsibilities of the data source v2 read API

2018-07-22 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-24882: External issue URL:

[jira] [Created] (SPARK-24882) separate responsibilities of the data source v2 read API

2018-07-22 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-24882: --- Summary: separate responsibilities of the data source v2 read API Key: SPARK-24882 URL: https://issues.apache.org/jira/browse/SPARK-24882 Project: Spark Issue

[jira] [Commented] (SPARK-24811) Add function `from_avro` and `to_avro`

2018-07-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551997#comment-16551997 ] Apache Spark commented on SPARK-24811: -- User 'gengliangwang' has created a pull request for this

[jira] [Commented] (SPARK-21274) Implement EXCEPT ALL and INTERSECT ALL

2018-07-22 Thread Dilip Biswal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551968#comment-16551968 ] Dilip Biswal commented on SPARK-21274: -- [~maropu] Thanks a lot for the info. > Implement EXCEPT

[jira] [Commented] (SPARK-16203) regexp_extract to return an ArrayType(StringType())

2018-07-22 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551966#comment-16551966 ] Herman van Hovell commented on SPARK-16203: --- [~nnicolini] adding {{regexp_extract_all}} makes

[jira] [Comment Edited] (SPARK-21274) Implement EXCEPT ALL and INTERSECT ALL

2018-07-22 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551965#comment-16551965 ] Takeshi Yamamuro edited comment on SPARK-21274 at 7/22/18 9:35 AM: ---

[jira] [Commented] (SPARK-21274) Implement EXCEPT ALL and INTERSECT ALL

2018-07-22 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551965#comment-16551965 ] Takeshi Yamamuro commented on SPARK-21274: -- ok, thanks! > Implement EXCEPT ALL and INTERSECT

[jira] [Commented] (SPARK-21274) Implement EXCEPT ALL and INTERSECT ALL

2018-07-22 Thread Dilip Biswal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551954#comment-16551954 ] Dilip Biswal commented on SPARK-21274: -- [~maropu] Hi Takeshi, yeah.. So the code that does the

[jira] [Assigned] (SPARK-24881) New options - compression and compressionLevel

2018-07-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24881: Assignee: Apache Spark > New options - compression and compressionLevel >

[jira] [Commented] (SPARK-24881) New options - compression and compressionLevel

2018-07-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551952#comment-16551952 ] Apache Spark commented on SPARK-24881: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24881) New options - compression and compressionLevel

2018-07-22 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24881: Assignee: (was: Apache Spark) > New options - compression and compressionLevel >

[jira] [Created] (SPARK-24881) New options - compression and compressionLevel

2018-07-22 Thread Maxim Gekk (JIRA)
Maxim Gekk created SPARK-24881: -- Summary: New options - compression and compressionLevel Key: SPARK-24881 URL: https://issues.apache.org/jira/browse/SPARK-24881 Project: Spark Issue Type:

[jira] [Commented] (SPARK-21274) Implement EXCEPT ALL and INTERSECT ALL

2018-07-22 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551921#comment-16551921 ] Takeshi Yamamuro commented on SPARK-21274: -- Anybody still working on this? > Implement EXCEPT