[jira] [Commented] (SPARK-24152) Flaky Test: SparkR

2018-05-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461965#comment-16461965 ] Hyukjin Kwon commented on SPARK-24152: -- Thanks [~viirya], I retriggered one build. Will resolve this

[jira] [Assigned] (SPARK-24152) Flaky Test: SparkR

2018-05-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-24152: Assignee: Liang-Chi Hsieh > Flaky Test: SparkR > -- > >

[jira] [Commented] (SPARK-24152) Flaky Test: SparkR

2018-05-02 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461962#comment-16461962 ] Liang-Chi Hsieh commented on SPARK-24152: - CRAN sysadmin replied me it should be fixed now. I

[jira] [Commented] (SPARK-24151) CURRENT_DATE, CURRENT_TIMESTAMP incorrectly resolved as column names when caseSensitive is enabled

2018-05-02 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461957#comment-16461957 ] Dongjoon Hyun commented on SPARK-24151: --- Thank you for reporting and fixing this, [~jamesthomp]. I

[jira] [Assigned] (SPARK-24169) JsonToStructs should not access SQLConf at executor side

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24169: Assignee: Apache Spark (was: Wenchen Fan) > JsonToStructs should not access SQLConf at

[jira] [Assigned] (SPARK-24169) JsonToStructs should not access SQLConf at executor side

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24169: Assignee: Wenchen Fan (was: Apache Spark) > JsonToStructs should not access SQLConf at

[jira] [Commented] (SPARK-24169) JsonToStructs should not access SQLConf at executor side

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461954#comment-16461954 ] Apache Spark commented on SPARK-24169: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Updated] (SPARK-24151) CURRENT_DATE, CURRENT_TIMESTAMP incorrectly resolved as column names when caseSensitive is enabled

2018-05-02 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-24151: -- Affects Version/s: 2.2.1 > CURRENT_DATE, CURRENT_TIMESTAMP incorrectly resolved as column

[jira] [Created] (SPARK-24169) JsonToStructs should not access SQLConf at executor side

2018-05-02 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-24169: --- Summary: JsonToStructs should not access SQLConf at executor side Key: SPARK-24169 URL: https://issues.apache.org/jira/browse/SPARK-24169 Project: Spark Issue

[jira] [Commented] (SPARK-24168) WindowExec should not access SQLConf at executor side

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461951#comment-16461951 ] Apache Spark commented on SPARK-24168: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24168) WindowExec should not access SQLConf at executor side

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24168: Assignee: Wenchen Fan (was: Apache Spark) > WindowExec should not access SQLConf at

[jira] [Assigned] (SPARK-24168) WindowExec should not access SQLConf at executor side

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24168: Assignee: Apache Spark (was: Wenchen Fan) > WindowExec should not access SQLConf at

[jira] [Created] (SPARK-24168) WindowExec should not access SQLConf at executor side

2018-05-02 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-24168: --- Summary: WindowExec should not access SQLConf at executor side Key: SPARK-24168 URL: https://issues.apache.org/jira/browse/SPARK-24168 Project: Spark Issue

[jira] [Assigned] (SPARK-24167) ParquetFilters should not access SQLConf at executor side

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24167: Assignee: Apache Spark (was: Wenchen Fan) > ParquetFilters should not access SQLConf at

[jira] [Assigned] (SPARK-24167) ParquetFilters should not access SQLConf at executor side

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24167: Assignee: Wenchen Fan (was: Apache Spark) > ParquetFilters should not access SQLConf at

[jira] [Commented] (SPARK-24167) ParquetFilters should not access SQLConf at executor side

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461949#comment-16461949 ] Apache Spark commented on SPARK-24167: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Created] (SPARK-24167) ParquetFilters should not access SQLConf at executor side

2018-05-02 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-24167: --- Summary: ParquetFilters should not access SQLConf at executor side Key: SPARK-24167 URL: https://issues.apache.org/jira/browse/SPARK-24167 Project: Spark

[jira] [Commented] (SPARK-24166) InMemoryTableScanExec should not access SQLConf at executor side

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461935#comment-16461935 ] Apache Spark commented on SPARK-24166: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24166) InMemoryTableScanExec should not access SQLConf at executor side

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24166: Assignee: Apache Spark (was: Wenchen Fan) > InMemoryTableScanExec should not access

[jira] [Assigned] (SPARK-24166) InMemoryTableScanExec should not access SQLConf at executor side

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24166: Assignee: Wenchen Fan (was: Apache Spark) > InMemoryTableScanExec should not access

[jira] [Created] (SPARK-24166) InMemoryTableScanExec should not access SQLConf at executor side

2018-05-02 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-24166: --- Summary: InMemoryTableScanExec should not access SQLConf at executor side Key: SPARK-24166 URL: https://issues.apache.org/jira/browse/SPARK-24166 Project: Spark

[jira] [Comment Edited] (SPARK-24152) Flaky Test: SparkR

2018-05-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461932#comment-16461932 ] Hyukjin Kwon edited comment on SPARK-24152 at 5/3/18 4:44 AM: -- For the past

[jira] [Commented] (SPARK-24152) Flaky Test: SparkR

2018-05-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461932#comment-16461932 ] Hyukjin Kwon commented on SPARK-24152: -- For the past issue, it was fixed within only a couple of

[jira] [Commented] (SPARK-24152) Flaky Test: SparkR

2018-05-02 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461931#comment-16461931 ] Shivaram Venkataraman commented on SPARK-24152: --- If this is blocking all PRs I think its

[jira] [Comment Edited] (SPARK-24152) Flaky Test: SparkR

2018-05-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461929#comment-16461929 ] Hyukjin Kwon edited comment on SPARK-24152 at 5/3/18 4:40 AM: -- >From

[jira] [Comment Edited] (SPARK-24152) Flaky Test: SparkR

2018-05-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461929#comment-16461929 ] Hyukjin Kwon edited comment on SPARK-24152 at 5/3/18 4:39 AM: -- >From

[jira] [Commented] (SPARK-24152) Flaky Test: SparkR

2018-05-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461929#comment-16461929 ] Hyukjin Kwon commented on SPARK-24152: -- >From Liang-Chi's comment and given previous discussion and

[jira] [Commented] (SPARK-24152) Flaky Test: SparkR

2018-05-02 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461920#comment-16461920 ] Shivaram Venkataraman commented on SPARK-24152: --- Unfortunately I dont have time to look at

[jira] [Commented] (SPARK-24152) Flaky Test: SparkR

2018-05-02 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461867#comment-16461867 ] Liang-Chi Hsieh commented on SPARK-24152: - Thanks [~hyukjin.kwon] for pinging me. I found a

[jira] [Assigned] (SPARK-24161) Enable debug package feature on structured streaming

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24161: Assignee: (was: Apache Spark) > Enable debug package feature on structured streaming

[jira] [Assigned] (SPARK-24161) Enable debug package feature on structured streaming

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24161: Assignee: Apache Spark > Enable debug package feature on structured streaming >

[jira] [Commented] (SPARK-24161) Enable debug package feature on structured streaming

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461858#comment-16461858 ] Apache Spark commented on SPARK-24161: -- User 'HeartSaVioR' has created a pull request for this

[jira] [Resolved] (SPARK-24110) Avoid calling UGI loginUserFromKeytab in ThriftServer

2018-05-02 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-24110. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21178

[jira] [Assigned] (SPARK-24110) Avoid calling UGI loginUserFromKeytab in ThriftServer

2018-05-02 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-24110: --- Assignee: Saisai Shao > Avoid calling UGI loginUserFromKeytab in ThriftServer >

[jira] [Commented] (SPARK-24152) Flaky Test: SparkR

2018-05-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461798#comment-16461798 ] Hyukjin Kwon commented on SPARK-24152: -- cc [~viirya] too > Flaky Test: SparkR > --

[jira] [Assigned] (SPARK-22812) Failing cran-check on master

2018-05-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-22812: Assignee: Liang-Chi Hsieh > Failing cran-check on master > -

[jira] [Created] (SPARK-24165) UDF within when().otherwise() raises NullPointerException

2018-05-02 Thread Jingxuan Wang (JIRA)
Jingxuan Wang created SPARK-24165: - Summary: UDF within when().otherwise() raises NullPointerException Key: SPARK-24165 URL: https://issues.apache.org/jira/browse/SPARK-24165 Project: Spark

[jira] [Commented] (SPARK-23429) Add executor memory metrics to heartbeat and expose in executors REST API

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461752#comment-16461752 ] Apache Spark commented on SPARK-23429: -- User 'edwinalu' has created a pull request for this issue:

[jira] [Created] (SPARK-24164) Support column list as the pivot column in Pivot

2018-05-02 Thread Maryann Xue (JIRA)
Maryann Xue created SPARK-24164: --- Summary: Support column list as the pivot column in Pivot Key: SPARK-24164 URL: https://issues.apache.org/jira/browse/SPARK-24164 Project: Spark Issue Type:

[jira] [Created] (SPARK-24163) Support "ANY" or sub-query for Pivot "IN" clause

2018-05-02 Thread Maryann Xue (JIRA)
Maryann Xue created SPARK-24163: --- Summary: Support "ANY" or sub-query for Pivot "IN" clause Key: SPARK-24163 URL: https://issues.apache.org/jira/browse/SPARK-24163 Project: Spark Issue Type:

[jira] [Created] (SPARK-24162) Support aliased literal values for Pivot "IN" clause

2018-05-02 Thread Maryann Xue (JIRA)
Maryann Xue created SPARK-24162: --- Summary: Support aliased literal values for Pivot "IN" clause Key: SPARK-24162 URL: https://issues.apache.org/jira/browse/SPARK-24162 Project: Spark Issue

[jira] [Resolved] (SPARK-24111) Add TPCDS v2.7 (latest) queries in TPCDSQueryBenchmark

2018-05-02 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-24111. - Resolution: Fixed Assignee: Takeshi Yamamuro Fix Version/s: 2.4.0 > Add TPCDS v2.7

[jira] [Created] (SPARK-24161) Enable debug package feature on structured streaming

2018-05-02 Thread Jungtaek Lim (JIRA)
Jungtaek Lim created SPARK-24161: Summary: Enable debug package feature on structured streaming Key: SPARK-24161 URL: https://issues.apache.org/jira/browse/SPARK-24161 Project: Spark Issue

[jira] [Commented] (SPARK-24161) Enable debug package feature on structured streaming

2018-05-02 Thread Jungtaek Lim (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461713#comment-16461713 ] Jungtaek Lim commented on SPARK-24161: -- I have a working patch. Will raise a PR sooner. > Enable

[jira] [Commented] (SPARK-24157) Enable no-data micro batches for streaming aggregation and deduplication

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461680#comment-16461680 ] Apache Spark commented on SPARK-24157: -- User 'tdas' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24157) Enable no-data micro batches for streaming aggregation and deduplication

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24157: Assignee: Apache Spark (was: Tathagata Das) > Enable no-data micro batches for streaming

[jira] [Assigned] (SPARK-24157) Enable no-data micro batches for streaming aggregation and deduplication

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24157: Assignee: Tathagata Das (was: Apache Spark) > Enable no-data micro batches for streaming

[jira] [Assigned] (SPARK-24160) ShuffleBlockFetcherIterator should fail if it receives zero-size blocks

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24160: Assignee: Josh Rosen (was: Apache Spark) > ShuffleBlockFetcherIterator should fail if it

[jira] [Assigned] (SPARK-24160) ShuffleBlockFetcherIterator should fail if it receives zero-size blocks

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24160: Assignee: Apache Spark (was: Josh Rosen) > ShuffleBlockFetcherIterator should fail if it

[jira] [Commented] (SPARK-24160) ShuffleBlockFetcherIterator should fail if it receives zero-size blocks

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461675#comment-16461675 ] Apache Spark commented on SPARK-24160: -- User 'JoshRosen' has created a pull request for this issue:

[jira] [Created] (SPARK-24160) ShuffleBlockFetcherIterator should fail if it receives zero-size blocks

2018-05-02 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-24160: -- Summary: ShuffleBlockFetcherIterator should fail if it receives zero-size blocks Key: SPARK-24160 URL: https://issues.apache.org/jira/browse/SPARK-24160 Project: Spark

[jira] [Updated] (SPARK-24155) Instrumentation improvement for clustering

2018-05-02 Thread Lu Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lu Wang updated SPARK-24155: Summary: Instrumentation improvement for clustering (was: Instrument improvement for clustering) >

[jira] [Updated] (SPARK-24132) Instrumentation improvement for classification

2018-05-02 Thread Lu Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lu Wang updated SPARK-24132: Summary: Instrumentation improvement for classification (was: Instruments improvement for classification)

[jira] [Created] (SPARK-24159) Enable no-data micro batches for streaming mapGroupswithState

2018-05-02 Thread Tathagata Das (JIRA)
Tathagata Das created SPARK-24159: - Summary: Enable no-data micro batches for streaming mapGroupswithState Key: SPARK-24159 URL: https://issues.apache.org/jira/browse/SPARK-24159 Project: Spark

[jira] [Created] (SPARK-24158) Enable no-data micro batches for streaming joins

2018-05-02 Thread Tathagata Das (JIRA)
Tathagata Das created SPARK-24158: - Summary: Enable no-data micro batches for streaming joins Key: SPARK-24158 URL: https://issues.apache.org/jira/browse/SPARK-24158 Project: Spark Issue

[jira] [Assigned] (SPARK-24158) Enable no-data micro batches for streaming joins

2018-05-02 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das reassigned SPARK-24158: - Assignee: Tathagata Das > Enable no-data micro batches for streaming joins >

[jira] [Created] (SPARK-24156) Enable no-data micro batches for more eager streaming state clean up

2018-05-02 Thread Tathagata Das (JIRA)
Tathagata Das created SPARK-24156: - Summary: Enable no-data micro batches for more eager streaming state clean up Key: SPARK-24156 URL: https://issues.apache.org/jira/browse/SPARK-24156 Project:

[jira] [Created] (SPARK-24157) Enable no-data micro batches for streaming aggregation and deduplication

2018-05-02 Thread Tathagata Das (JIRA)
Tathagata Das created SPARK-24157: - Summary: Enable no-data micro batches for streaming aggregation and deduplication Key: SPARK-24157 URL: https://issues.apache.org/jira/browse/SPARK-24157 Project:

[jira] [Assigned] (SPARK-24155) Instrument improvement for clustering

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24155: Assignee: Apache Spark > Instrument improvement for clustering >

[jira] [Commented] (SPARK-24155) Instrument improvement for clustering

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461640#comment-16461640 ] Apache Spark commented on SPARK-24155: -- User 'ludatabricks' has created a pull request for this

[jira] [Assigned] (SPARK-24155) Instrument improvement for clustering

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24155: Assignee: (was: Apache Spark) > Instrument improvement for clustering >

[jira] [Created] (SPARK-24155) Instrument improvement for clustering

2018-05-02 Thread Lu Wang (JIRA)
Lu Wang created SPARK-24155: --- Summary: Instrument improvement for clustering Key: SPARK-24155 URL: https://issues.apache.org/jira/browse/SPARK-24155 Project: Spark Issue Type: Sub-task

[jira] [Resolved] (SPARK-18791) Stream-Stream Joins

2018-05-02 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-18791. --- Resolution: Done Fix Version/s: 2.3.0 > Stream-Stream Joins > --- > >

[jira] [Resolved] (SPARK-23923) High-order function: cardinality(x) → bigint

2018-05-02 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-23923. - Resolution: Fixed Assignee: Kazuaki Ishizaki Fix Version/s: 2.4.0 > High-order function:

[jira] [Resolved] (SPARK-24123) Fix a flaky test `DateTimeUtilsSuite.monthsBetween`

2018-05-02 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-24123. - Resolution: Fixed Assignee: Marco Gaido Fix Version/s: 2.4.0 > Fix a flaky test

[jira] [Updated] (SPARK-24097) Instruments improvements - RandomForest and GradientBoostedTree

2018-05-02 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-24097: -- Shepherd: Joseph K. Bradley > Instruments improvements - RandomForest and

[jira] [Assigned] (SPARK-24097) Instruments improvements - RandomForest and GradientBoostedTree

2018-05-02 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley reassigned SPARK-24097: - Assignee: Weichen Xu > Instruments improvements - RandomForest and

[jira] [Resolved] (SPARK-24133) Reading Parquet files containing large strings can fail with java.lang.ArrayIndexOutOfBoundsException

2018-05-02 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-24133. - Resolution: Fixed Assignee: Ala Luszczak Fix Version/s: 2.4.0 > Reading Parquet files

[jira] [Updated] (SPARK-24154) AccumulatorV2 loses type information during serialization

2018-05-02 Thread Sergey Zhemzhitsky (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Zhemzhitsky updated SPARK-24154: --- Description: AccumulatorV2 loses type information during serialization. It happens

[jira] [Created] (SPARK-24154) AccumulatorV2 loses type information during serialization

2018-05-02 Thread Sergey Zhemzhitsky (JIRA)
Sergey Zhemzhitsky created SPARK-24154: -- Summary: AccumulatorV2 loses type information during serialization Key: SPARK-24154 URL: https://issues.apache.org/jira/browse/SPARK-24154 Project: Spark

[jira] [Commented] (SPARK-4502) Spark SQL reads unneccesary nested fields from Parquet

2018-05-02 Thread Evan McClain (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461502#comment-16461502 ] Evan McClain commented on SPARK-4502: - The workaround I've been using is to explicitly pass in the

[jira] [Updated] (SPARK-23971) Should not leak Spark sessions across test suites

2018-05-02 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23971: Fix Version/s: 2.3.1 > Should not leak Spark sessions across test suites >

[jira] [Updated] (SPARK-23971) Should not leak Spark sessions across test suites

2018-05-02 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23971: Component/s: Tests > Should not leak Spark sessions across test suites >

[jira] [Resolved] (SPARK-24013) ApproximatePercentile grinds to a halt on sorted input.

2018-05-02 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-24013. - Resolution: Fixed Assignee: Marco Gaido Fix Version/s: 2.4.0 > ApproximatePercentile

[jira] [Updated] (SPARK-23489) Flaky Test: HiveExternalCatalogVersionsSuite

2018-05-02 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-23489: -- Description: I saw this error in an unrelated PR. It seems to me a bad configuration in the

[jira] [Created] (SPARK-24153) Flaky Test: DirectKafkaStreamSuite

2018-05-02 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-24153: - Summary: Flaky Test: DirectKafkaStreamSuite Key: SPARK-24153 URL: https://issues.apache.org/jira/browse/SPARK-24153 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-24152) Flaky Test: SparkR

2018-05-02 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-24152: -- Description: PR builder and master branch test fails with the following SparkR error with

[jira] [Updated] (SPARK-24152) Flaky Test: SparkR

2018-05-02 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-24152: -- Description: PR builder and master branch test fails with the following SparkR error with

[jira] [Updated] (SPARK-24152) Flaky Test: SparkR

2018-05-02 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-24152: -- Description: PR builder fails with the following SparkR error with unknown reason. The

[jira] [Commented] (SPARK-24152) Flaky Test: SparkR

2018-05-02 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461328#comment-16461328 ] Dongjoon Hyun commented on SPARK-24152: --- cc [~shivaram], [~felixcheung], [~yanboliang] > Flaky

[jira] [Updated] (SPARK-24152) Flaky Test: SparkR

2018-05-02 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-24152: -- Description: PR builder fails with the following SparkR error with unknown reason. The

[jira] [Created] (SPARK-24152) Flaky Test: SparkR

2018-05-02 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-24152: - Summary: Flaky Test: SparkR Key: SPARK-24152 URL: https://issues.apache.org/jira/browse/SPARK-24152 Project: Spark Issue Type: Bug Components:

[jira] [Commented] (SPARK-24135) [K8s] Executors that fail to start up because of init-container errors are not retried and limit the executor pool size

2018-05-02 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461324#comment-16461324 ] Erik Erlandson commented on SPARK-24135: > In the case of the executor failing to start at all,

[jira] [Updated] (SPARK-24150) Race condition in FsHistoryProvider

2018-05-02 Thread William Montaz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Montaz updated SPARK-24150: --- Description: There exist a race condition in checkLogs method between threads of

[jira] [Updated] (SPARK-24150) Race condition in FsHistoryProvider

2018-05-02 Thread William Montaz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Montaz updated SPARK-24150: --- Description: There exist a race condition in checkLogs method between threads of

[jira] [Updated] (SPARK-24150) Race condition in FsHistoryProvider

2018-05-02 Thread William Montaz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Montaz updated SPARK-24150: --- Priority: Major (was: Minor) > Race condition in FsHistoryProvider >

[jira] [Updated] (SPARK-24150) Race condition in FsHistoryProvider

2018-05-02 Thread William Montaz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Montaz updated SPARK-24150: --- Description: There exist a race condition in checkLogs method between threads of

[jira] [Updated] (SPARK-24150) Race condition in FsHistoryProvider

2018-05-02 Thread William Montaz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Montaz updated SPARK-24150: --- Description: There exist a race condition in checkLogs method between threads of

[jira] [Updated] (SPARK-24150) Race condition in FsHistoryProvider

2018-05-02 Thread William Montaz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Montaz updated SPARK-24150: --- Description: There exist a race condition in checkLogs method between threads of

[jira] [Commented] (SPARK-22918) sbt test (spark - local) fail after upgrading to 2.2.1 with: java.security.AccessControlException: access denied org.apache.derby.security.SystemPermission( "engine",

2018-05-02 Thread Sam Garrett (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461250#comment-16461250 ] Sam Garrett commented on SPARK-22918: - +1 same issue > sbt test (spark - local) fail after upgrading

[jira] [Assigned] (SPARK-24151) CURRENT_DATE, CURRENT_TIMESTAMP incorrectly resolved as column names when caseSensitive is enabled

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24151: Assignee: (was: Apache Spark) > CURRENT_DATE, CURRENT_TIMESTAMP incorrectly resolved

[jira] [Assigned] (SPARK-24151) CURRENT_DATE, CURRENT_TIMESTAMP incorrectly resolved as column names when caseSensitive is enabled

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24151: Assignee: Apache Spark > CURRENT_DATE, CURRENT_TIMESTAMP incorrectly resolved as column

[jira] [Commented] (SPARK-24151) CURRENT_DATE, CURRENT_TIMESTAMP incorrectly resolved as column names when caseSensitive is enabled

2018-05-02 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461237#comment-16461237 ] Apache Spark commented on SPARK-24151: -- User 'jamesthomp' has created a pull request for this issue:

[jira] [Created] (SPARK-24151) CURRENT_DATE, CURRENT_TIMESTAMP incorrectly resolved as column names when caseSensitive is enabled

2018-05-02 Thread James Thompson (JIRA)
James Thompson created SPARK-24151: -- Summary: CURRENT_DATE, CURRENT_TIMESTAMP incorrectly resolved as column names when caseSensitive is enabled Key: SPARK-24151 URL:

[jira] [Comment Edited] (SPARK-24135) [K8s] Executors that fail to start up because of init-container errors are not retried and limit the executor pool size

2018-05-02 Thread Matt Cheah (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461188#comment-16461188 ] Matt Cheah edited comment on SPARK-24135 at 5/2/18 3:37 PM: {quote}Restarting

[jira] [Comment Edited] (SPARK-24135) [K8s] Executors that fail to start up because of init-container errors are not retried and limit the executor pool size

2018-05-02 Thread Matt Cheah (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460047#comment-16460047 ] Matt Cheah edited comment on SPARK-24135 at 5/2/18 3:37 PM: {quote}But I'm

[jira] [Commented] (SPARK-24135) [K8s] Executors that fail to start up because of init-container errors are not retried and limit the executor pool size

2018-05-02 Thread Matt Cheah (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461188#comment-16461188 ] Matt Cheah commented on SPARK-24135: > Restarting seems like it would eventually be limited by the

[jira] [Updated] (SPARK-24150) Race condition in FsHistoryProvider

2018-05-02 Thread William Montaz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Montaz updated SPARK-24150: --- Description: There exist a race condition between the method checkLogs and cleanLogs.

[jira] [Created] (SPARK-24150) Race condition in FsHistoryProvider

2018-05-02 Thread William Montaz (JIRA)
William Montaz created SPARK-24150: -- Summary: Race condition in FsHistoryProvider Key: SPARK-24150 URL: https://issues.apache.org/jira/browse/SPARK-24150 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-24135) [K8s] Executors that fail to start up because of init-container errors are not retried and limit the executor pool size

2018-05-02 Thread Erik Erlandson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461154#comment-16461154 ] Erik Erlandson commented on SPARK-24135: IIRC the dynamic allocation heuristic was to avoid

  1   2   >