[jira] [Created] (SPARK-24529) Add spotbugs into maven build process

2018-06-12 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-24529: Summary: Add spotbugs into maven build process Key: SPARK-24529 URL: https://issues.apache.org/jira/browse/SPARK-24529 Project: Spark Issue Type:

[jira] [Created] (SPARK-24528) Missing optimization for Aggregations/Windowing on a bucketed table

2018-06-12 Thread Ohad Raviv (JIRA)
Ohad Raviv created SPARK-24528: -- Summary: Missing optimization for Aggregations/Windowing on a bucketed table Key: SPARK-24528 URL: https://issues.apache.org/jira/browse/SPARK-24528 Project: Spark

[jira] [Updated] (SPARK-24474) Cores are left idle when there are a lot of tasks to run

2018-06-12 Thread Al M (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Al M updated SPARK-24474: - Summary: Cores are left idle when there are a lot of tasks to run (was: Cores are left idle when there are a

[jira] [Created] (SPARK-24527) select column alias should support quotation marks

2018-06-12 Thread ice bai (JIRA)
ice bai created SPARK-24527: --- Summary: select column alias should support quotation marks Key: SPARK-24527 URL: https://issues.apache.org/jira/browse/SPARK-24527 Project: Spark Issue Type:

[jira] [Updated] (SPARK-24505) Convert strings in codegen to blocks: Cast and BoundAttribute

2018-06-12 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-24505: Summary: Convert strings in codegen to blocks: Cast and BoundAttribute (was: Forbidding

[jira] [Commented] (SPARK-24505) Forbidding string interpolation in CodeBlock

2018-06-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509362#comment-16509362 ] Apache Spark commented on SPARK-24505: -- User 'viirya' has created a pull request for this issue:

[jira] [Updated] (SPARK-24505) Convert strings in codegen to blocks: Cast and BoundAttribute

2018-06-12 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-24505: Description: The CodeBlock interpolator now accepts strings. Based on previous

[jira] [Commented] (SPARK-23754) StopIterator exception in Python UDF results in partial result

2018-06-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509442#comment-16509442 ] Apache Spark commented on SPARK-23754: -- User 'e-dorigatti' has created a pull request for this

[jira] [Assigned] (SPARK-24500) UnsupportedOperationException when trying to execute Union plan with Stream of children

2018-06-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24500: Assignee: Herman van Hovell (was: Apache Spark) > UnsupportedOperationException when

[jira] [Commented] (SPARK-24500) UnsupportedOperationException when trying to execute Union plan with Stream of children

2018-06-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509453#comment-16509453 ] Apache Spark commented on SPARK-24500: -- User 'hvanhovell' has created a pull request for this

[jira] [Assigned] (SPARK-24500) UnsupportedOperationException when trying to execute Union plan with Stream of children

2018-06-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24500: Assignee: Apache Spark (was: Herman van Hovell) > UnsupportedOperationException when

[jira] [Commented] (SPARK-24528) Missing optimization for Aggregations/Windowing on a bucketed table

2018-06-12 Thread Ohad Raviv (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509465#comment-16509465 ] Ohad Raviv commented on SPARK-24528: [~cloud_fan], [~viirya] - Hi I found somewhat similar issue to

[jira] [Commented] (SPARK-24529) Add spotbugs into maven build process

2018-06-12 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509569#comment-16509569 ] Kazuaki Ishizaki commented on SPARK-24529: -- I am working for this > Add spotbugs into maven

[jira] [Updated] (SPARK-24530) pyspark.ml doesn't generate class docs correctly

2018-06-12 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24530: -- Attachment: Screen Shot 2018-06-12 at 8.23.29 AM.png > pyspark.ml doesn't generate class docs

[jira] [Created] (SPARK-24531) HiveExternalCatalogVersionsSuite failing due to missing 2.2.0 version

2018-06-12 Thread Marco Gaido (JIRA)
Marco Gaido created SPARK-24531: --- Summary: HiveExternalCatalogVersionsSuite failing due to missing 2.2.0 version Key: SPARK-24531 URL: https://issues.apache.org/jira/browse/SPARK-24531 Project: Spark

[jira] [Updated] (SPARK-24530) pyspark.ml doesn't generate class docs correctly

2018-06-12 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24530: -- Attachment: Screen Shot 2018-06-12 at 8.23.18 AM.png > pyspark.ml doesn't generate class docs

[jira] [Updated] (SPARK-24530) pyspark.ml doesn't generate class docs correctly

2018-06-12 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24530: -- Priority: Blocker (was: Major) > pyspark.ml doesn't generate class docs correctly >

[jira] [Closed] (SPARK-24112) Add `spark.sql.hive.convertMetastoreTableProperty` for backward compatiblility

2018-06-12 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-24112. - > Add `spark.sql.hive.convertMetastoreTableProperty` for backward compatiblility >

[jira] [Updated] (SPARK-24531) HiveExternalCatalogVersionsSuite failing due to missing 2.2.0 version

2018-06-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-24531: --- Target Version/s: 2.2.2, 2.3.2 > HiveExternalCatalogVersionsSuite failing due to missing

[jira] [Resolved] (SPARK-15064) Locale support in StopWordsRemover

2018-06-12 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-15064. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21501

[jira] [Created] (SPARK-24530) pyspark.ml doesn't generate class docs correctly

2018-06-12 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-24530: - Summary: pyspark.ml doesn't generate class docs correctly Key: SPARK-24530 URL: https://issues.apache.org/jira/browse/SPARK-24530 Project: Spark Issue

[jira] [Updated] (SPARK-24530) pyspark.ml doesn't generate class docs correctly

2018-06-12 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24530: -- Description: I generated python docs from master locally using `make html`. However, the

[jira] [Assigned] (SPARK-24531) HiveExternalCatalogVersionsSuite failing due to missing 2.2.0 version

2018-06-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24531: Assignee: (was: Apache Spark) > HiveExternalCatalogVersionsSuite failing due to

[jira] [Commented] (SPARK-24531) HiveExternalCatalogVersionsSuite failing due to missing 2.2.0 version

2018-06-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509748#comment-16509748 ] Apache Spark commented on SPARK-24531: -- User 'mgaido91' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24531) HiveExternalCatalogVersionsSuite failing due to missing 2.2.0 version

2018-06-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24531: Assignee: Apache Spark > HiveExternalCatalogVersionsSuite failing due to missing 2.2.0

[jira] [Created] (SPARK-24532) HiveExternalCatalogVersionSuite should be resilient to missing versions

2018-06-12 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-24532: -- Summary: HiveExternalCatalogVersionSuite should be resilient to missing versions Key: SPARK-24532 URL: https://issues.apache.org/jira/browse/SPARK-24532 Project:

[jira] [Updated] (SPARK-24531) HiveExternalCatalogVersionsSuite failing due to missing 2.2.0 version

2018-06-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-24531: --- Priority: Blocker (was: Major) > HiveExternalCatalogVersionsSuite failing due to missing

[jira] [Commented] (SPARK-15064) Locale support in StopWordsRemover

2018-06-12 Thread Lee Dongjin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509799#comment-16509799 ] Lee Dongjin commented on SPARK-15064: - [~mengxr] Hello. Please assign this issue to me. > Locale

[jira] [Resolved] (SPARK-24416) Update configuration definition for spark.blacklist.killBlacklistedExecutors

2018-06-12 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-24416. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21475

[jira] [Commented] (SPARK-24531) HiveExternalCatalogVersionsSuite failing due to missing 2.2.0 version

2018-06-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510072#comment-16510072 ] Apache Spark commented on SPARK-24531: -- User 'mgaido91' has created a pull request for this issue:

[jira] [Updated] (SPARK-24534) Add a way to bypass entrypoint.sh script if no spark cmd is passed

2018-06-12 Thread Ricardo Martinelli de Oliveira (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ricardo Martinelli de Oliveira updated SPARK-24534: --- Description: As an improvement in the entrypoint.sh script,

[jira] [Commented] (SPARK-24258) SPIP: Improve PySpark support for ML Matrix and Vector types

2018-06-12 Thread Leif Walsh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510406#comment-16510406 ] Leif Walsh commented on SPARK-24258: I think for PySpark users, we could just make it easy to use

[jira] [Created] (SPARK-24537) Add array_remove / array_zip / map_from_arrays / array_distinct

2018-06-12 Thread Huaxin Gao (JIRA)
Huaxin Gao created SPARK-24537: -- Summary: Add array_remove / array_zip / map_from_arrays / array_distinct Key: SPARK-24537 URL: https://issues.apache.org/jira/browse/SPARK-24537 Project: Spark

[jira] [Commented] (SPARK-24537) Add array_remove / array_zip / map_from_arrays / array_distinct

2018-06-12 Thread Huaxin Gao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510418#comment-16510418 ] Huaxin Gao commented on SPARK-24537: I will work on this. Thanks! > Add array_remove / array_zip /

[jira] [Resolved] (SPARK-22239) User-defined window functions with pandas udf

2018-06-12 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-22239. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21082

[jira] [Assigned] (SPARK-22239) User-defined window functions with pandas udf

2018-06-12 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-22239: Assignee: Li Jin > User-defined window functions with pandas udf >

[jira] [Created] (SPARK-24539) HistoryServer does not display metrics from tasks that complete after stage failure

2018-06-12 Thread Imran Rashid (JIRA)
Imran Rashid created SPARK-24539: Summary: HistoryServer does not display metrics from tasks that complete after stage failure Key: SPARK-24539 URL: https://issues.apache.org/jira/browse/SPARK-24539

[jira] [Commented] (SPARK-24538) Decimal type support push down to the data sources

2018-06-12 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510503#comment-16510503 ] Yuming Wang commented on SPARK-24538: - I'm working on this. > Decimal type support push down to the

[jira] [Created] (SPARK-24538) Decimal type support push down to the data sources

2018-06-12 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-24538: --- Summary: Decimal type support push down to the data sources Key: SPARK-24538 URL: https://issues.apache.org/jira/browse/SPARK-24538 Project: Spark Issue Type:

[jira] [Commented] (SPARK-24538) Decimal type support push down to the data sources

2018-06-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510508#comment-16510508 ] Apache Spark commented on SPARK-24538: -- User 'wangyum' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24538) Decimal type support push down to the data sources

2018-06-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24538: Assignee: (was: Apache Spark) > Decimal type support push down to the data sources >

[jira] [Issue Comment Deleted] (SPARK-24538) Decimal type support push down to the data sources

2018-06-12 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-24538: Comment: was deleted (was: I'm working on this.) > Decimal type support push down to the data

[jira] [Assigned] (SPARK-24538) Decimal type support push down to the data sources

2018-06-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24538: Assignee: Apache Spark > Decimal type support push down to the data sources >

[jira] [Resolved] (SPARK-24466) TextSocketMicroBatchReader no longer works with nc utility

2018-06-12 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24466. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21497

[jira] [Assigned] (SPARK-24466) TextSocketMicroBatchReader no longer works with nc utility

2018-06-12 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-24466: Assignee: Jungtaek Lim > TextSocketMicroBatchReader no longer works with nc utility >

[jira] [Updated] (SPARK-24536) Query with nonsensical LIMIT hits AssertionError

2018-06-12 Thread Alexander Behm (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Behm updated SPARK-24536: --- Labels: beginner (was: ) > Query with nonsensical LIMIT hits AssertionError >

[jira] [Created] (SPARK-24536) Query with nonsensical LIMIT hits AssertionError

2018-06-12 Thread Alexander Behm (JIRA)
Alexander Behm created SPARK-24536: -- Summary: Query with nonsensical LIMIT hits AssertionError Key: SPARK-24536 URL: https://issues.apache.org/jira/browse/SPARK-24536 Project: Spark Issue

[jira] [Commented] (SPARK-22357) SparkContext.binaryFiles ignore minPartitions parameter

2018-06-12 Thread John Brock (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510507#comment-16510507 ] John Brock commented on SPARK-22357: What are people's opinions of [~bomeng]'s fix? This bug just

[jira] [Updated] (SPARK-24538) Decimal type support push down to the data sources

2018-06-12 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-24538: Description: Latest parquet support decimal type statistics. then we can push down: {noformat}

[jira] [Updated] (SPARK-24538) Decimal type support push down to the data sources

2018-06-12 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-24538: Description: Latest parquet support decimal type statistics. then we can push down to the data

[jira] [Updated] (SPARK-24530) pyspark.ml doesn't generate class docs correctly

2018-06-12 Thread Lee Dongjin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lee Dongjin updated SPARK-24530: Attachment: pyspark-ml-doc-utuntu18.04-python2.7-sphinx-1.7.5.png > pyspark.ml doesn't generate

[jira] [Commented] (SPARK-24530) pyspark.ml doesn't generate class docs correctly

2018-06-12 Thread Lee Dongjin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510570#comment-16510570 ] Lee Dongjin commented on SPARK-24530: - In my case, it works correctly on current master (commit

[jira] [Resolved] (SPARK-23606) Flakey FileBasedDataSourceSuite

2018-06-12 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-23606. --- Resolution: Duplicate Thank you for reporting, [~henryr] . This is already tracked by

[jira] [Updated] (SPARK-24540) Support for multiple delimiter in Spark CSV read

2018-06-12 Thread Ashwin K (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin K updated SPARK-24540: - Description: Currently, the delimiter option Spark 2.0 to read and split CSV files/data only support a

[jira] [Created] (SPARK-24540) Support for multiple delimiter in Spark CSV read

2018-06-12 Thread Ashwin K (JIRA)
Ashwin K created SPARK-24540: Summary: Support for multiple delimiter in Spark CSV read Key: SPARK-24540 URL: https://issues.apache.org/jira/browse/SPARK-24540 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-24485) Measure and log elapsed time for filesystem operations in HDFSBackedStateStoreProvider

2018-06-12 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24485. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21506

[jira] [Assigned] (SPARK-24485) Measure and log elapsed time for filesystem operations in HDFSBackedStateStoreProvider

2018-06-12 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-24485: Assignee: Jungtaek Lim > Measure and log elapsed time for filesystem operations in >

[jira] [Updated] (SPARK-20592) Alter table concatenate is not working as expected.

2018-06-12 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-20592: -- Affects Version/s: 2.2.1 2.3.1 > Alter table concatenate is not

[jira] [Updated] (SPARK-24536) Query with nonsensical LIMIT hits AssertionError

2018-06-12 Thread Alexander Behm (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Behm updated SPARK-24536: --- Description: SELECT COUNT(1) FROM t LIMIT CAST(NULL AS INT) fails in the QueryPlanner

[jira] [Resolved] (SPARK-24506) Spark.ui.filters not applied to /sqlserver/ url

2018-06-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24506. Resolution: Fixed Assignee: Marco Gaido Fix Version/s: 2.4.0

[jira] [Commented] (SPARK-23797) SparkSQL performance on small TPCDS tables is very low when compared to Drill or Presto

2018-06-12 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510612#comment-16510612 ] Dongjoon Hyun commented on SPARK-23797: --- In general, I agree with [~maropu] . This is too general

[jira] [Created] (SPARK-24541) TCP based shuffle

2018-06-12 Thread Jose Torres (JIRA)
Jose Torres created SPARK-24541: --- Summary: TCP based shuffle Key: SPARK-24541 URL: https://issues.apache.org/jira/browse/SPARK-24541 Project: Spark Issue Type: Sub-task Components:

[jira] [Assigned] (SPARK-23933) High-order function: map(array, array) → map

2018-06-12 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin reassigned SPARK-23933: - Assignee: Kazuaki Ishizaki > High-order function: map(array, array) → map >

[jira] [Commented] (SPARK-23030) Decrease memory consumption with toPandas() collection using Arrow

2018-06-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510235#comment-16510235 ] Apache Spark commented on SPARK-23030: -- User 'BryanCutler' has created a pull request for this

[jira] [Assigned] (SPARK-23030) Decrease memory consumption with toPandas() collection using Arrow

2018-06-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23030: Assignee: Apache Spark > Decrease memory consumption with toPandas() collection using

[jira] [Assigned] (SPARK-23030) Decrease memory consumption with toPandas() collection using Arrow

2018-06-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23030: Assignee: (was: Apache Spark) > Decrease memory consumption with toPandas()

[jira] [Resolved] (SPARK-23933) High-order function: map(array, array) → map

2018-06-12 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-23933. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21258

[jira] [Created] (SPARK-24534) Add a way to bypass entrypoint.sh script if no spark cmd is passed

2018-06-12 Thread Ricardo Martinelli de Oliveira (JIRA)
Ricardo Martinelli de Oliveira created SPARK-24534: -- Summary: Add a way to bypass entrypoint.sh script if no spark cmd is passed Key: SPARK-24534 URL:

[jira] [Updated] (SPARK-24534) Add a way to bypass entrypoint.sh script if no spark cmd is passed

2018-06-12 Thread Ricardo Martinelli de Oliveira (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ricardo Martinelli de Oliveira updated SPARK-24534: --- Description: As an improvement in the entrypoint.sh script,

[jira] [Commented] (SPARK-23010) Add integration testing for Kubernetes backend into the apache/spark repository

2018-06-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510095#comment-16510095 ] Apache Spark commented on SPARK-23010: -- User 'jiangxb1987' has created a pull request for this

[jira] [Created] (SPARK-24535) Fix java version parsing in SparkR

2018-06-12 Thread Shivaram Venkataraman (JIRA)
Shivaram Venkataraman created SPARK-24535: - Summary: Fix java version parsing in SparkR Key: SPARK-24535 URL: https://issues.apache.org/jira/browse/SPARK-24535 Project: Spark Issue

[jira] [Commented] (SPARK-23435) R tests should support latest testthat

2018-06-12 Thread Weiqiang Zhuang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510065#comment-16510065 ] Weiqiang Zhuang commented on SPARK-23435: - I hit this since I just reinstalled R and bunch of

[jira] [Commented] (SPARK-23931) High-order function: array_zip(array1, array2[, ...]) → array

2018-06-12 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510079#comment-16510079 ] Marco Gaido commented on SPARK-23931: - I just edited the title/description in order to update to the

[jira] [Updated] (SPARK-23931) High-order function: array_zip(array1, array2[, ...]) → array

2018-06-12 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido updated SPARK-23931: Description: Ref: https://prestodb.io/docs/current/functions/array.html Merges the given arrays,

[jira] [Resolved] (SPARK-24216) Spark TypedAggregateExpression uses getSimpleName that is not safe in scala

2018-06-12 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-24216. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21276

[jira] [Assigned] (SPARK-24216) Spark TypedAggregateExpression uses getSimpleName that is not safe in scala

2018-06-12 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-24216: --- Assignee: Fangshi Li > Spark TypedAggregateExpression uses getSimpleName that is not safe

[jira] [Assigned] (SPARK-24416) Update configuration definition for spark.blacklist.killBlacklistedExecutors

2018-06-12 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-24416: Assignee: Sanket Reddy > Update configuration definition for

[jira] [Assigned] (SPARK-23931) High-order function: zip(array1, array2[, ...]) → array

2018-06-12 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin reassigned SPARK-23931: - Assignee: Dylan Guedes > High-order function: zip(array1, array2[, ...]) → array >

[jira] [Updated] (SPARK-23931) High-order function: array_zip(array1, array2[, ...]) → array

2018-06-12 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido updated SPARK-23931: Summary: High-order function: array_zip(array1, array2[, ...]) → array (was: High-order

[jira] [Resolved] (SPARK-23931) High-order function: zip(array1, array2[, ...]) → array

2018-06-12 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-23931. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21045

[jira] [Commented] (SPARK-20168) Enable kinesis to start stream from Initial position specified by a timestamp

2018-06-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509933#comment-16509933 ] Apache Spark commented on SPARK-20168: -- User 'yashs360' has created a pull request for this issue:

[jira] [Reopened] (SPARK-20168) Enable kinesis to start stream from Initial position specified by a timestamp

2018-06-12 Thread Yash Sharma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yash Sharma reopened SPARK-20168: - The last patch in the kinesis streaming receiver sets the timestamp for the mode AT_TIMESTAMP, but

[jira] [Assigned] (SPARK-20168) Enable kinesis to start stream from Initial position specified by a timestamp

2018-06-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20168: Assignee: Apache Spark (was: Yash Sharma) > Enable kinesis to start stream from Initial

[jira] [Assigned] (SPARK-20168) Enable kinesis to start stream from Initial position specified by a timestamp

2018-06-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20168: Assignee: Yash Sharma (was: Apache Spark) > Enable kinesis to start stream from Initial

[jira] [Commented] (SPARK-24529) Add spotbugs into maven build process

2018-06-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509949#comment-16509949 ] Apache Spark commented on SPARK-24529: -- User 'kiszk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-24529) Add spotbugs into maven build process

2018-06-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24529: Assignee: (was: Apache Spark) > Add spotbugs into maven build process >

[jira] [Assigned] (SPARK-24529) Add spotbugs into maven build process

2018-06-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24529: Assignee: Apache Spark > Add spotbugs into maven build process >

[jira] [Commented] (SPARK-14376) spark.ml parity for trees

2018-06-12 Thread Lee Dongjin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509658#comment-16509658 ] Lee Dongjin commented on SPARK-14376: - [~josephkb] Excuse me. Is there any reason this issue is

[jira] [Commented] (SPARK-24005) Remove usage of Scala’s parallel collection

2018-06-12 Thread Maxim Gekk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509667#comment-16509667 ] Maxim Gekk commented on SPARK-24005: [~smilegator] I am trying to reproduce the issue but so far I

[jira] [Resolved] (SPARK-24481) GeneratedIteratorForCodegenStage1 grows beyond 64 KB

2018-06-12 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido resolved SPARK-24481. - Resolution: Not A Problem I am resolving this as no further action can be taken IMHO (other

[jira] [Comment Edited] (SPARK-24481) GeneratedIteratorForCodegenStage1 grows beyond 64 KB

2018-06-12 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509662#comment-16509662 ] Marco Gaido edited comment on SPARK-24481 at 6/12/18 2:03 PM: -- I am

[jira] [Comment Edited] (SPARK-24476) java.net.SocketTimeoutException: Read timed out under jets3t while running the Spark Structured Streaming

2018-06-12 Thread bharath kumar avusherla (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509708#comment-16509708 ] bharath kumar avusherla edited comment on SPARK-24476 at 6/12/18 2:48 PM:

[jira] [Commented] (SPARK-24476) java.net.SocketTimeoutException: Read timed out under jets3t while running the Spark Structured Streaming

2018-06-12 Thread bharath kumar avusherla (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509708#comment-16509708 ] bharath kumar avusherla commented on SPARK-24476: - Our initial try was enable the

[jira] [Created] (SPARK-24533) typesafe has rebranded to lightbend. change the build/mvn endpoint from downloads.typesafe.com to downloads.lightbend.com

2018-06-12 Thread Sanket Reddy (JIRA)
Sanket Reddy created SPARK-24533: Summary: typesafe has rebranded to lightbend. change the build/mvn endpoint from downloads.typesafe.com to downloads.lightbend.com Key: SPARK-24533 URL:

[jira] [Commented] (SPARK-24533) typesafe has rebranded to lightbend. change the build/mvn endpoint from downloads.typesafe.com to downloads.lightbend.com

2018-06-12 Thread Sanket Reddy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509850#comment-16509850 ] Sanket Reddy commented on SPARK-24533: -- I will put up a PR shortly thanks > typesafe has rebranded