[jira] [Commented] (SPARK-24707) Enable spark-kafka-streaming to maintain min buffer using async thread to avoid blocking kafka poll

[ https://issues.apache.org/jira/browse/SPARK-24707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16528967#comment-16528967 ] Apache Spark commented on SPARK-24707: -- User 'sidhavratha' has created a pull reque

[jira] [Assigned] (SPARK-24707) Enable spark-kafka-streaming to maintain min buffer using async thread to avoid blocking kafka poll

[ https://issues.apache.org/jira/browse/SPARK-24707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24707: Assignee: (was: Apache Spark) > Enable spark-kafka-streaming to maintain min buffer u

[jira] [Assigned] (SPARK-24707) Enable spark-kafka-streaming to maintain min buffer using async thread to avoid blocking kafka poll

[ https://issues.apache.org/jira/browse/SPARK-24707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24707: Assignee: Apache Spark > Enable spark-kafka-streaming to maintain min buffer using async

[jira] [Updated] (SPARK-24707) Enable spark-kafka-streaming to maintain min buffer using async thread to avoid blocking kafka poll

[ https://issues.apache.org/jira/browse/SPARK-24707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidhavratha Kumar updated SPARK-24707: -- Component/s: (was: Structured Streaming) DStreams > Enable spark-

[jira] [Updated] (SPARK-24707) Enable spark-kafka-streaming to maintain min buffer using async thread to avoid blocking kafka poll

[ https://issues.apache.org/jira/browse/SPARK-24707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidhavratha Kumar updated SPARK-24707: -- Description: Currently Spark Kafka RDD will block on kafka consumer poll. Specially in

[jira] [Updated] (SPARK-24707) Enable spark-kafka-streaming to maintain min buffer using async thread to avoid blocking kafka poll

[ https://issues.apache.org/jira/browse/SPARK-24707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sidhavratha Kumar updated SPARK-24707: -- Attachment: 40_partition_topic_without_buffer.pdf > Enable spark-kafka-streaming to ma

[jira] [Created] (SPARK-24707) Enable spark-kafka-streaming to maintain min buffer using async thread to avoid blocking kafka poll

Sidhavratha Kumar created SPARK-24707: - Summary: Enable spark-kafka-streaming to maintain min buffer using async thread to avoid blocking kafka poll Key: SPARK-24707 URL: https://issues.apache.org/jira/browse/

[jira] [Assigned] (SPARK-24470) RestSubmissionClient to be robust against 404 & non json responses

[ https://issues.apache.org/jira/browse/SPARK-24470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24470: Assignee: (was: Apache Spark) > RestSubmissionClient to be robust against 404 & non j

[jira] [Commented] (SPARK-24470) RestSubmissionClient to be robust against 404 & non json responses

[ https://issues.apache.org/jira/browse/SPARK-24470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16528955#comment-16528955 ] Apache Spark commented on SPARK-24470: -- User 'rekhajoshm' has created a pull reques

[jira] [Assigned] (SPARK-24470) RestSubmissionClient to be robust against 404 & non json responses

[ https://issues.apache.org/jira/browse/SPARK-24470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24470: Assignee: Apache Spark > RestSubmissionClient to be robust against 404 & non json respons

[jira] [Resolved] (SPARK-24654) Update, fix LICENSE and NOTICE, and specialize for source vs binary

[ https://issues.apache.org/jira/browse/SPARK-24654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-24654. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21640 [https://github.c

[jira] [Commented] (SPARK-24507) Description in "Level of Parallelism in Data Receiving" section of Spark Streaming Programming Guide in is not relevan for the recent Kafka direct apprach

[ https://issues.apache.org/jira/browse/SPARK-24507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16528928#comment-16528928 ] Apache Spark commented on SPARK-24507: -- User 'rekhajoshm' has created a pull reques

[jira] [Assigned] (SPARK-24507) Description in "Level of Parallelism in Data Receiving" section of Spark Streaming Programming Guide in is not relevan for the recent Kafka direct apprach

[ https://issues.apache.org/jira/browse/SPARK-24507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24507: Assignee: Apache Spark > Description in "Level of Parallelism in Data Receiving" section

[jira] [Assigned] (SPARK-24507) Description in "Level of Parallelism in Data Receiving" section of Spark Streaming Programming Guide in is not relevan for the recent Kafka direct apprach

[ https://issues.apache.org/jira/browse/SPARK-24507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24507: Assignee: (was: Apache Spark) > Description in "Level of Parallelism in Data Receivin

[jira] [Commented] (SPARK-17181) [Spark2.0 web ui]The status of the certain jobs is still displayed as running even if all the stages of this job have already finished

[ https://issues.apache.org/jira/browse/SPARK-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16528919#comment-16528919 ] Ron Kitay commented on SPARK-17181: --- [~tgraves] - This seems like a very old issue, ho

[jira] [Updated] (SPARK-21443) Very long planning duration for queries with lots of operations

[ https://issues.apache.org/jira/browse/SPARK-21443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rekha Joshi updated SPARK-21443: Description: Creating a streaming query with large amount of operations and fields (100+) results

[jira] [Commented] (SPARK-24706) Support ByteType and ShortType pushdown to parquet

[ https://issues.apache.org/jira/browse/SPARK-24706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16528878#comment-16528878 ] Yuming Wang commented on SPARK-24706: - Benchmark result: {noformat} ###

[jira] [Updated] (SPARK-24706) Support ByteType and ShortType pushdown to parquet

[ https://issues.apache.org/jira/browse/SPARK-24706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-24706: Description: (was: Benchmark result: {noformat} ###[ Pushdown benc

[jira] [Updated] (SPARK-24706) Support ByteType and ShortType pushdown to parquet

[ https://issues.apache.org/jira/browse/SPARK-24706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-24706: Description: Benchmark result: {noformat} ###[ Pushdown benchmark for

[jira] [Assigned] (SPARK-24706) Support ByteType and ShortType pushdown to parquet

[ https://issues.apache.org/jira/browse/SPARK-24706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24706: Assignee: Apache Spark > Support ByteType and ShortType pushdown to parquet > ---

[jira] [Assigned] (SPARK-24706) Support ByteType and ShortType pushdown to parquet

[ https://issues.apache.org/jira/browse/SPARK-24706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24706: Assignee: (was: Apache Spark) > Support ByteType and ShortType pushdown to parquet >

[jira] [Commented] (SPARK-24706) Support ByteType and ShortType pushdown to parquet

[ https://issues.apache.org/jira/browse/SPARK-24706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16528866#comment-16528866 ] Apache Spark commented on SPARK-24706: -- User 'wangyum' has created a pull request f

[jira] [Created] (SPARK-24706) Support ByteType and ShortType pushdown to parquet

Yuming Wang created SPARK-24706: --- Summary: Support ByteType and ShortType pushdown to parquet Key: SPARK-24706 URL: https://issues.apache.org/jira/browse/SPARK-24706 Project: Spark Issue Type:

[jira] [Updated] (SPARK-24705) Spark.sql.adaptive.enabled=true is enabled and self-join query

[ https://issues.apache.org/jira/browse/SPARK-24705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cheng updated SPARK-24705: -- Description: [~smilegator] When loading data using jdbc and enabling spark.sql.adaptive.enabled=true , for ex

[jira] [Updated] (SPARK-24705) Spark.sql.adaptive.enabled=true is enabled and self-join query

[ https://issues.apache.org/jira/browse/SPARK-24705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cheng updated SPARK-24705: -- Description: When loading data using jdbc and enabling spark.sql.adaptive.enabled=true , for example loadin

[jira] [Updated] (SPARK-24705) Spark.sql.adaptive.enabled=true is enabled and self-join query

[ https://issues.apache.org/jira/browse/SPARK-24705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cheng updated SPARK-24705: -- Description: When loading data using jdbc and enabling spark.sql.adaptive.enabled=true , for example loadin

[jira] [Updated] (SPARK-24705) Spark.sql.adaptive.enabled=true is enabled and self-join query

[ https://issues.apache.org/jira/browse/SPARK-24705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cheng updated SPARK-24705: -- Description: When loading data using jdbc and enabling spark.sql.adaptive.enabled=true , for example loadin

[jira] [Updated] (SPARK-24705) Spark.sql.adaptive.enabled=true is enabled and self-join query

[ https://issues.apache.org/jira/browse/SPARK-24705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cheng updated SPARK-24705: -- Attachment: Error stack.txt > Spark.sql.adaptive.enabled=true is enabled and self-join query > ---

[jira] [Issue Comment Deleted] (SPARK-24705) Spark.sql.adaptive.enabled=true is enabled and self-join query

[ https://issues.apache.org/jira/browse/SPARK-24705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cheng updated SPARK-24705: -- Comment: was deleted (was: For example, my query: device_loc table comes from the jdbc data source select tv_

[jira] [Updated] (SPARK-24705) Spark.sql.adaptive.enabled=true is enabled and self-join query

[ https://issues.apache.org/jira/browse/SPARK-24705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cheng updated SPARK-24705: -- Affects Version/s: 2.2.1 > Spark.sql.adaptive.enabled=true is enabled and self-join query > --

[jira] [Updated] (SPARK-24705) Spark.sql.adaptive.enabled=true is enabled and self-join query

[ https://issues.apache.org/jira/browse/SPARK-24705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cheng updated SPARK-24705: -- Description: When loading data using jdbc and enabling spark.sql.adaptive.enabled=true , for example loadin

[jira] [Commented] (SPARK-24705) Spark.sql.adaptive.enabled=true is enabled and self-join query

[ https://issues.apache.org/jira/browse/SPARK-24705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16528711#comment-16528711 ] cheng commented on SPARK-24705: --- For example, my query: device_loc table comes from the jd

[jira] [Created] (SPARK-24705) Spark.sql.adaptive.enabled=true is enabled and self-join query

cheng created SPARK-24705: - Summary: Spark.sql.adaptive.enabled=true is enabled and self-join query Key: SPARK-24705 URL: https://issues.apache.org/jira/browse/SPARK-24705 Project: Spark Issue Type:

[jira] [Commented] (SPARK-23725) Improve Hadoop's LineReader to support charsets different from UTF-8

[ https://issues.apache.org/jira/browse/SPARK-23725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16528691#comment-16528691 ] Maxim Gekk commented on SPARK-23725: [~hyukjin.kwon] I am working on the implementat

[jira] [Commented] (SPARK-24530) Sphinx doesn't render autodoc_docstring_signature correctly (with Python 2?) and pyspark.ml docs are broken

[ https://issues.apache.org/jira/browse/SPARK-24530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16528629#comment-16528629 ] Hyukjin Kwon commented on SPARK-24530: -- [~vanzin] and [~jerryshao] FYI. > Sphinx d

[jira] [Assigned] (SPARK-24704) The order of stages in the DAG graph is incorrect

[ https://issues.apache.org/jira/browse/SPARK-24704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24704: Assignee: Apache Spark > The order of stages in the DAG graph is incorrect >

[jira] [Assigned] (SPARK-24704) The order of stages in the DAG graph is incorrect

[ https://issues.apache.org/jira/browse/SPARK-24704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24704: Assignee: (was: Apache Spark) > The order of stages in the DAG graph is incorrect > -

[jira] [Commented] (SPARK-24704) The order of stages in the DAG graph is incorrect

[ https://issues.apache.org/jira/browse/SPARK-24704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16528625#comment-16528625 ] Apache Spark commented on SPARK-24704: -- User 'stanzhai' has created a pull request

[jira] [Updated] (SPARK-24704) The order of stages in the DAG graph is incorrect

[ https://issues.apache.org/jira/browse/SPARK-24704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] StanZhai updated SPARK-24704: - Attachment: WX20180630-161907.png > The order of stages in the DAG graph is incorrect >

[jira] [Created] (SPARK-24704) The order of stages in the DAG graph is incorrect

StanZhai created SPARK-24704: Summary: The order of stages in the DAG graph is incorrect Key: SPARK-24704 URL: https://issues.apache.org/jira/browse/SPARK-24704 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-23773) JacksonGenerator does not include keys that have null value for StructTypes

[ https://issues.apache.org/jira/browse/SPARK-23773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-23773. -- Resolution: Won't Fix Let me leave this resolved for now given the discussion in the PR but pl

[jira] [Assigned] (SPARK-24695) Unable to return calendar interval from udf

[ https://issues.apache.org/jira/browse/SPARK-24695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24695: Assignee: (was: Apache Spark) > Unable to return calendar interval from udf > ---

[jira] [Assigned] (SPARK-24695) Unable to return calendar interval from udf

[ https://issues.apache.org/jira/browse/SPARK-24695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24695: Assignee: Apache Spark > Unable to return calendar interval from udf > --

[jira] [Commented] (SPARK-24695) Unable to return calendar interval from udf

[ https://issues.apache.org/jira/browse/SPARK-24695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16528597#comment-16528597 ] Apache Spark commented on SPARK-24695: -- User 'priyankagargnitk' has created a pull

[jira] [Resolved] (SPARK-24696) ColumnPruning rule fails to remove extra Project

[ https://issues.apache.org/jira/browse/SPARK-24696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-24696. - Resolution: Fixed Fix Version/s: 2.4.0 2.3.2 > ColumnPruning rule fails to rem