[jira] [Assigned] (SPARK-22347) UDF is evaluated when 'F.when' condition is false

2017-10-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22347: Assignee: (was: Apache Spark) > UDF is evaluated when 'F.when' condition is false >

[jira] [Assigned] (SPARK-22347) UDF is evaluated when 'F.when' condition is false

2017-10-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22347: Assignee: Apache Spark > UDF is evaluated when 'F.when' condition is false >

[jira] [Commented] (SPARK-22347) UDF is evaluated when 'F.when' condition is false

2017-10-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16221719#comment-16221719 ] Apache Spark commented on SPARK-22347: -- User 'viirya' has created a pull request for this issue:

[jira] [Comment Edited] (SPARK-22347) UDF is evaluated when 'F.when' condition is false

2017-10-26 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16221612#comment-16221612 ] Liang-Chi Hsieh edited comment on SPARK-22347 at 10/27/17 5:06 AM: ---

[jira] [Comment Edited] (SPARK-22347) UDF is evaluated when 'F.when' condition is false

2017-10-26 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16221612#comment-16221612 ] Liang-Chi Hsieh edited comment on SPARK-22347 at 10/27/17 5:05 AM: ---

[jira] [Commented] (SPARK-18755) Add Randomized Grid Search to Spark ML

2017-10-26 Thread Ilya Matiach (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16221687#comment-16221687 ] Ilya Matiach commented on SPARK-18755: -- I've created a PR that adds randomized grid search to

[jira] [Comment Edited] (SPARK-22347) UDF is evaluated when 'F.when' condition is false

2017-10-26 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16221612#comment-16221612 ] Liang-Chi Hsieh edited comment on SPARK-22347 at 10/27/17 2:44 AM: ---

[jira] [Commented] (SPARK-22347) UDF is evaluated when 'F.when' condition is false

2017-10-26 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16221612#comment-16221612 ] Liang-Chi Hsieh commented on SPARK-22347: - Under the current execution mode of Python UDFs, I

[jira] [Resolved] (SPARK-22355) Dataset.collect is not threadsafe

2017-10-26 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-22355. - Resolution: Fixed Fix Version/s: 2.3.0 2.2.1 > Dataset.collect is not

[jira] [Resolved] (SPARK-22356) data source table should support overlapped columns between data and partition schema

2017-10-26 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-22356. - Resolution: Fixed Fix Version/s: 2.3.0 2.2.1 > data source table should

[jira] [Resolved] (SPARK-22366) Support ignoreMissingFiles flag parallel to ignoreCorruptFiles

2017-10-26 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-22366. -- Resolution: Fixed Assignee: Jose Torres Fix Version/s: 2.3.0 > Support

[jira] [Commented] (SPARK-22344) Prevent R CMD check from using /tmp

2017-10-26 Thread Hossein Falaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16221460#comment-16221460 ] Hossein Falaki commented on SPARK-22344: I don't have solid pointer as to why we are creating

[jira] [Commented] (SPARK-22305) HDFSBackedStateStoreProvider fails with StackOverflowException when attempting to recover state

2017-10-26 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16221432#comment-16221432 ] Shixiong Zhu commented on SPARK-22305: -- [~Yuval.Itzchakov] how many batches per 1 minute in your

[jira] [Assigned] (SPARK-22339) Push epoch updates to executors on fetch failure to avoid fetch retries for missing executors

2017-10-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22339: Assignee: (was: Apache Spark) > Push epoch updates to executors on fetch failure to

[jira] [Commented] (SPARK-22339) Push epoch updates to executors on fetch failure to avoid fetch retries for missing executors

2017-10-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16221423#comment-16221423 ] Apache Spark commented on SPARK-22339: -- User 'juanrh' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22339) Push epoch updates to executors on fetch failure to avoid fetch retries for missing executors

2017-10-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22339: Assignee: Apache Spark > Push epoch updates to executors on fetch failure to avoid fetch

[jira] [Resolved] (SPARK-22131) Add Mesos Secrets Support to the Mesos Driver

2017-10-26 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-22131. Resolution: Fixed Assignee: Susan X. Huynh Fix Version/s: 2.3.0 > Add

[jira] [Commented] (SPARK-22357) SparkContext.binaryFiles ignore minPartitions parameter

2017-10-26 Thread Ilya Matiach (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16221388#comment-16221388 ] Ilya Matiach commented on SPARK-22357: -- binaryFiles ignores the number of partitions I want to have,

[jira] [Assigned] (SPARK-20644) Hook up Spark UI to the new key-value store backend

2017-10-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20644: Assignee: Apache Spark > Hook up Spark UI to the new key-value store backend >

[jira] [Assigned] (SPARK-20644) Hook up Spark UI to the new key-value store backend

2017-10-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20644: Assignee: (was: Apache Spark) > Hook up Spark UI to the new key-value store backend >

[jira] [Commented] (SPARK-20644) Hook up Spark UI to the new key-value store backend

2017-10-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16221372#comment-16221372 ] Apache Spark commented on SPARK-20644: -- User 'vanzin' has created a pull request for this issue:

[jira] [Updated] (SPARK-22366) Support ignoreMissingFiles flag parallel to ignoreCorruptFiles

2017-10-26 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-22366: - Component/s: (was: Spark Core) SQL > Support ignoreMissingFiles flag

[jira] [Updated] (SPARK-22366) Support ignoreMissingFiles flag parallel to ignoreCorruptFiles

2017-10-26 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-22366: - Issue Type: Improvement (was: Bug) > Support ignoreMissingFiles flag parallel to

[jira] [Commented] (SPARK-22366) Support ignoreMissingFiles flag parallel to ignoreCorruptFiles

2017-10-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16221225#comment-16221225 ] Apache Spark commented on SPARK-22366: -- User 'joseph-torres' has created a pull request for this

[jira] [Assigned] (SPARK-22366) Support ignoreMissingFiles flag parallel to ignoreCorruptFiles

2017-10-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22366: Assignee: Apache Spark > Support ignoreMissingFiles flag parallel to ignoreCorruptFiles >

[jira] [Assigned] (SPARK-22366) Support ignoreMissingFiles flag parallel to ignoreCorruptFiles

2017-10-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22366: Assignee: (was: Apache Spark) > Support ignoreMissingFiles flag parallel to

[jira] [Created] (SPARK-22366) Support ignoreMissingFiles flag parallel to ignoreCorruptFiles

2017-10-26 Thread Jose Torres (JIRA)
Jose Torres created SPARK-22366: --- Summary: Support ignoreMissingFiles flag parallel to ignoreCorruptFiles Key: SPARK-22366 URL: https://issues.apache.org/jira/browse/SPARK-22366 Project: Spark

[jira] [Resolved] (SPARK-22328) ClosureCleaner misses referenced superclass fields, gives them null values

2017-10-26 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-22328. - Resolution: Fixed Fix Version/s: 2.3.0 2.2.1 Issue resolved by pull

[jira] [Assigned] (SPARK-22328) ClosureCleaner misses referenced superclass fields, gives them null values

2017-10-26 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-22328: --- Assignee: Liang-Chi Hsieh > ClosureCleaner misses referenced superclass fields, gives them

[jira] [Commented] (SPARK-22357) SparkContext.binaryFiles ignore minPartitions parameter

2017-10-26 Thread Bo Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16221078#comment-16221078 ] Bo Meng commented on SPARK-22357: - a quick fix could be as follows, correct me if i am wrong. val

[jira] [Commented] (SPARK-20928) SPIP: Continuous Processing Mode for Structured Streaming

2017-10-26 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16221025#comment-16221025 ] Cody Koeninger commented on SPARK-20928: No, it doesn't exist yet as far as I know. Reason I ask

[jira] [Commented] (SPARK-22184) GraphX fails in case of insufficient memory and checkpoints enabled

2017-10-26 Thread Sergey Zhemzhitsky (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16221019#comment-16221019 ] Sergey Zhemzhitsky commented on SPARK-22184: Hello guys, is there a chance for this issue to

[jira] [Commented] (SPARK-22150) PeriodicCheckpointer fails with FileNotFoundException in case of dependant RDDs

2017-10-26 Thread Sergey Zhemzhitsky (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16221018#comment-16221018 ] Sergey Zhemzhitsky commented on SPARK-22150: Hello guys, is there a chance for this issue to

[jira] [Commented] (SPARK-20928) SPIP: Continuous Processing Mode for Structured Streaming

2017-10-26 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220956#comment-16220956 ] Reynold Xin commented on SPARK-20928: - That doesn't yet exist does it? How would that work for

[jira] [Commented] (SPARK-22344) Prevent R CMD check from using /tmp

2017-10-26 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220884#comment-16220884 ] Shivaram Venkataraman commented on SPARK-22344: --- Thanks for investigating. Is

[jira] [Commented] (SPARK-11334) numRunningTasks can't be less than 0, or it will affect executor allocation

2017-10-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220881#comment-16220881 ] Apache Spark commented on SPARK-11334: -- User 'sitalkedia' has created a pull request for this issue:

[jira] [Commented] (SPARK-22312) Spark job stuck with no executor due to bug in Executor Allocation Manager

2017-10-26 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220857#comment-16220857 ] Apache Spark commented on SPARK-22312: -- User 'sitalkedia' has created a pull request for this issue:

[jira] [Resolved] (SPARK-22364) unix_timestamp function sets valid dates to null

2017-10-26 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-22364. --- Resolution: Not A Problem These functions largely mimic Hive on purpose, and I think that's just how

[jira] [Commented] (SPARK-22042) ReorderJoinPredicates can break when child's partitioning is not decided

2017-10-26 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220734#comment-16220734 ] Andrew Ash commented on SPARK-22042: Hi I'm seeing this problem as well, thanks for investigating and

[jira] [Commented] (SPARK-22364) unix_timestamp function sets valid dates to null

2017-10-26 Thread Matthew Sinton-Hewitt (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220697#comment-16220697 ] Matthew Sinton-Hewitt commented on SPARK-22364: --- Yes I think you're right. However, it

[jira] [Commented] (SPARK-20928) SPIP: Continuous Processing Mode for Structured Streaming

2017-10-26 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220694#comment-16220694 ] Cody Koeninger commented on SPARK-20928: Can you clarify how this impacts sinks having access to

[jira] [Resolved] (SPARK-20643) Implement listener for saving application status data in key-value store

2017-10-26 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-20643. -- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19383

[jira] [Assigned] (SPARK-20643) Implement listener for saving application status data in key-value store

2017-10-26 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-20643: Assignee: Marcelo Vanzin > Implement listener for saving application status data in

[jira] [Commented] (SPARK-22365) Spark UI executors empty list with 500 error

2017-10-26 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220631#comment-16220631 ] Sean Owen commented on SPARK-22365: --- Is there no other error following this? a "Caused by" log? if it's

[jira] [Commented] (SPARK-22365) Spark UI executors empty list with 500 error

2017-10-26 Thread Jakub Dubovsky (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220630#comment-16220630 ] Jakub Dubovsky commented on SPARK-22365: Is there anything I can do to investigate? I am a BE

[jira] [Commented] (SPARK-22365) Spark UI executors empty list with 500 error

2017-10-26 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220627#comment-16220627 ] Sean Owen commented on SPARK-22365: --- This doesn't actually show the underlying error, just reporting

[jira] [Created] (SPARK-22365) Spark UI executors empty list with 500 error

2017-10-26 Thread Jakub Dubovsky (JIRA)
Jakub Dubovsky created SPARK-22365: -- Summary: Spark UI executors empty list with 500 error Key: SPARK-22365 URL: https://issues.apache.org/jira/browse/SPARK-22365 Project: Spark Issue Type:

[jira] [Commented] (SPARK-22364) unix_timestamp function sets valid dates to null

2017-10-26 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220483#comment-16220483 ] Sean Owen commented on SPARK-22364: --- Is the problem that this hour doesn't exist in the timezone

[jira] [Updated] (SPARK-22364) unix_timestamp function sets valid dates to null

2017-10-26 Thread Matthew Sinton-Hewitt (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Sinton-Hewitt updated SPARK-22364: -- Description: org.apache.spark.sql.functions.unix_timestamp sets some valid

[jira] [Updated] (SPARK-22364) unix_timestamp function sets valid dates to null

2017-10-26 Thread Matthew Sinton-Hewitt (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthew Sinton-Hewitt updated SPARK-22364: -- Description: org.apache.spark.sql.functions.unix_timestamp sets some valid

[jira] [Commented] (SPARK-21657) Spark has exponential time complexity to explode(array of structs)

2017-10-26 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220469#comment-16220469 ] Sean Owen commented on SPARK-21657: --- I suspect that something somewhere is doing something that's

[jira] [Created] (SPARK-22364) unix_timestamp function sets valid dates to null

2017-10-26 Thread Matthew Sinton-Hewitt (JIRA)
Matthew Sinton-Hewitt created SPARK-22364: - Summary: unix_timestamp function sets valid dates to null Key: SPARK-22364 URL: https://issues.apache.org/jira/browse/SPARK-22364 Project: Spark

[jira] [Commented] (SPARK-21657) Spark has exponential time complexity to explode(array of structs)

2017-10-26 Thread Ohad Raviv (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220450#comment-16220450 ] Ohad Raviv commented on SPARK-21657: Hi, Wanted to add that we're facing exactly the same issue. 6

[jira] [Commented] (SPARK-22357) SparkContext.binaryFiles ignore minPartitions parameter

2017-10-26 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220431#comment-16220431 ] Saisai Shao commented on SPARK-22357: - Yes, I know this parameter is ignored, but I'm not sure is it

[jira] [Commented] (SPARK-22357) SparkContext.binaryFiles ignore minPartitions parameter

2017-10-26 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220417#comment-16220417 ] Weichen Xu commented on SPARK-22357: [~jerryshao] I checked the code, it ignore the `minPartitions`

[jira] [Commented] (SPARK-22357) SparkContext.binaryFiles ignore minPartitions parameter

2017-10-26 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220395#comment-16220395 ] Saisai Shao commented on SPARK-22357: - bq. In our specific case this was a problem, because we

[jira] [Commented] (SPARK-22323) Design doc for different types of pandas_udf

2017-10-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220373#comment-16220373 ] Hyukjin Kwon commented on SPARK-22323: -- Let me leave a link to the discussion doc -

[jira] [Created] (SPARK-22363) Add unit test for Window spilling

2017-10-26 Thread Jiang Xingbo (JIRA)
Jiang Xingbo created SPARK-22363: Summary: Add unit test for Window spilling Key: SPARK-22363 URL: https://issues.apache.org/jira/browse/SPARK-22363 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-22362) Add unit test for Window Aggregate Functions

2017-10-26 Thread Jiang Xingbo (JIRA)
Jiang Xingbo created SPARK-22362: Summary: Add unit test for Window Aggregate Functions Key: SPARK-22362 URL: https://issues.apache.org/jira/browse/SPARK-22362 Project: Spark Issue Type:

[jira] [Created] (SPARK-22361) Add unit test for Window Frames

2017-10-26 Thread Jiang Xingbo (JIRA)
Jiang Xingbo created SPARK-22361: Summary: Add unit test for Window Frames Key: SPARK-22361 URL: https://issues.apache.org/jira/browse/SPARK-22361 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-22360) Add unit test for Window Specifications

2017-10-26 Thread Jiang Xingbo (JIRA)
Jiang Xingbo created SPARK-22360: Summary: Add unit test for Window Specifications Key: SPARK-22360 URL: https://issues.apache.org/jira/browse/SPARK-22360 Project: Spark Issue Type: Sub-task

[jira] [Assigned] (SPARK-17902) collect() ignores stringsAsFactors

2017-10-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-17902: Assignee: Hyukjin Kwon > collect() ignores stringsAsFactors >

[jira] [Resolved] (SPARK-17902) collect() ignores stringsAsFactors

2017-10-26 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-17902. -- Resolution: Fixed Fix Version/s: 2.1.3 2.3.0 2.2.1

[jira] [Created] (SPARK-22359) Improve the test coverage of window functions

2017-10-26 Thread Jiang Xingbo (JIRA)
Jiang Xingbo created SPARK-22359: Summary: Improve the test coverage of window functions Key: SPARK-22359 URL: https://issues.apache.org/jira/browse/SPARK-22359 Project: Spark Issue Type:

[jira] [Commented] (SPARK-22358) ClassNotFoundException at NettyRpcEnv#deserialize when running Spark on Yarn-Cluster

2017-10-26 Thread Congxian Qiu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220280#comment-16220280 ] Congxian Qiu commented on SPARK-22358: -- I run the same program many times, got the above error some

[jira] [Comment Edited] (SPARK-9686) Spark Thrift server doesn't return correct JDBC metadata

2017-10-26 Thread Andriy Kushnir (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16217201#comment-16217201 ] Andriy Kushnir edited comment on SPARK-9686 at 10/26/17 10:18 AM: --

[jira] [Closed] (SPARK-22358) ClassNotFoundException at NettyRpcEnv#deserialize when running Spark on Yarn-Cluster

2017-10-26 Thread Congxian Qiu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Congxian Qiu closed SPARK-22358. > ClassNotFoundException at NettyRpcEnv#deserialize when running Spark on > Yarn-Cluster >

[jira] [Resolved] (SPARK-22358) ClassNotFoundException at NettyRpcEnv#deserialize when running Spark on Yarn-Cluster

2017-10-26 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-22358. --- Resolution: Cannot Reproduce You have some strange corruption in your code or build: RpcAddvess You

[jira] [Created] (SPARK-22358) ClassNotFoundException at NettyRpcEnv#deserialize when running Spark on Yarn-Cluster

2017-10-26 Thread Congxian Qiu (JIRA)
Congxian Qiu created SPARK-22358: Summary: ClassNotFoundException at NettyRpcEnv#deserialize when running Spark on Yarn-Cluster Key: SPARK-22358 URL: https://issues.apache.org/jira/browse/SPARK-22358

[jira] [Commented] (SPARK-21725) spark thriftserver insert overwrite table partition select

2017-10-26 Thread xinzhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220239#comment-16220239 ] xinzhang commented on SPARK-21725: -- I tried the spark(version-master) at 21/Aug2017, it still appear

[jira] [Commented] (SPARK-21725) spark thriftserver insert overwrite table partition select

2017-10-26 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220228#comment-16220228 ] Marco Gaido commented on SPARK-21725: - please try with the master branch, not with Spark 2.1.2. I

[jira] [Commented] (SPARK-21725) spark thriftserver insert overwrite table partition select

2017-10-26 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220225#comment-16220225 ] Sean Owen commented on SPARK-21725: --- [~zhangxin0112zx] there's no reason to expect 2.1.2 was different.

[jira] [Commented] (SPARK-21725) spark thriftserver insert overwrite table partition select

2017-10-26 Thread xinzhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220216#comment-16220216 ] xinzhang commented on SPARK-21725: -- I download spark 2.1.2 .The problem still appear . Could u give me

[jira] [Comment Edited] (SPARK-15905) Driver hung while writing to console progress bar

2017-10-26 Thread Denis Gabaydulin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220167#comment-16220167 ] Denis Gabaydulin edited comment on SPARK-15905 at 10/26/17 8:48 AM:

[jira] [Commented] (SPARK-22357) SparkContext.binaryFiles ignore minPartitions parameter

2017-10-26 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220169#comment-16220169 ] Weichen Xu commented on SPARK-22357: [~jerryshao] code formatted. This bug is reported from here

[jira] [Comment Edited] (SPARK-15905) Driver hung while writing to console progress bar

2017-10-26 Thread Denis Gabaydulin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220167#comment-16220167 ] Denis Gabaydulin edited comment on SPARK-15905 at 10/26/17 8:42 AM:

[jira] [Updated] (SPARK-22357) SparkContext.binaryFiles ignore minPartitions parameter

2017-10-26 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu updated SPARK-22357: --- Description: this is a bug in binaryFiles - even though we give it the partitions, binaryFiles

[jira] [Commented] (SPARK-15905) Driver hung while writing to console progress bar

2017-10-26 Thread Denis Gabaydulin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220167#comment-16220167 ] Denis Gabaydulin commented on SPARK-15905: -- Not sure I've got the same issue. But I have at

[jira] [Commented] (SPARK-22357) SparkContext.binaryFiles ignore minPartitions parameter

2017-10-26 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220160#comment-16220160 ] Saisai Shao commented on SPARK-22357: - [~WeichenXu123] would you please format the code in JIRA

[jira] [Commented] (SPARK-22357) SparkContext.binaryFiles ignore minPartitions parameter

2017-10-26 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220152#comment-16220152 ] Weichen Xu commented on SPARK-22357: cc [~jerryshao] > SparkContext.binaryFiles ignore minPartitions

[jira] [Commented] (SPARK-22357) SparkContext.binaryFiles ignore minPartitions parameter

2017-10-26 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220147#comment-16220147 ] Weichen Xu commented on SPARK-22357: cc [~imatiach] [~liancheng] > SparkContext.binaryFiles ignore

[jira] [Created] (SPARK-22357) SparkContext.binaryFiles ignore minPartitions parameter

2017-10-26 Thread Weichen Xu (JIRA)
Weichen Xu created SPARK-22357: -- Summary: SparkContext.binaryFiles ignore minPartitions parameter Key: SPARK-22357 URL: https://issues.apache.org/jira/browse/SPARK-22357 Project: Spark Issue

[jira] [Assigned] (SPARK-21840) Allow multiple SparkSubmit invocations in same JVM without polluting system properties

2017-10-26 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-21840: --- Assignee: Marcelo Vanzin > Allow multiple SparkSubmit invocations in same JVM without

[jira] [Resolved] (SPARK-21840) Allow multiple SparkSubmit invocations in same JVM without polluting system properties

2017-10-26 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-21840. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19519

[jira] [Resolved] (SPARK-22308) Support unit tests of spark code using ScalaTest using suites other than FunSuite

2017-10-26 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-22308. - Resolution: Fixed Assignee: Nathan Kronenfeld Fix Version/s: 2.3.0 > Support unit tests

[jira] [Commented] (SPARK-22344) Prevent R CMD check from using /tmp

2017-10-26 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220083#comment-16220083 ] Felix Cheung commented on SPARK-22344: -- why is hive there when enableHiveSupport should be off? what

[jira] [Comment Edited] (SPARK-22344) Prevent R CMD check from using /tmp

2017-10-26 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220080#comment-16220080 ] Felix Cheung edited comment on SPARK-22344 at 10/26/17 7:25 AM: this is

[jira] [Commented] (SPARK-22344) Prevent R CMD check from using /tmp

2017-10-26 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220080#comment-16220080 ] Felix Cheung commented on SPARK-22344: -- this is what I see on a clean machine tracking