[jira] [Assigned] (SPARK-33765) Migrate UNCACHE TABLE to new resolution framework

[ https://issues.apache.org/jira/browse/SPARK-33765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33765: Assignee: (was: Apache Spark) > Migrate UNCACHE TABLE to new resolution framework >

[jira] [Commented] (SPARK-33765) Migrate UNCACHE TABLE to new resolution framework

[ https://issues.apache.org/jira/browse/SPARK-33765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17248283#comment-17248283 ] Apache Spark commented on SPARK-33765: -- User 'imback82' has created a pull request for this issue:

[jira] [Assigned] (SPARK-33765) Migrate UNCACHE TABLE to new resolution framework

[ https://issues.apache.org/jira/browse/SPARK-33765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33765: Assignee: Apache Spark > Migrate UNCACHE TABLE to new resolution framework >

[jira] [Created] (SPARK-33765) Migrate UNCACHE TABLE to new resolution framework

Terry Kim created SPARK-33765: - Summary: Migrate UNCACHE TABLE to new resolution framework Key: SPARK-33765 URL: https://issues.apache.org/jira/browse/SPARK-33765 Project: Spark Issue Type:

[jira] [Commented] (SPARK-33762) Bump commons-codec to latest version.

[ https://issues.apache.org/jira/browse/SPARK-33762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17248273#comment-17248273 ] Apache Spark commented on SPARK-33762: -- User 'n-marion' has created a pull request for this issue:

[jira] [Assigned] (SPARK-33762) Bump commons-codec to latest version.

[ https://issues.apache.org/jira/browse/SPARK-33762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33762: Assignee: Apache Spark > Bump commons-codec to latest version. >

[jira] [Assigned] (SPARK-33762) Bump commons-codec to latest version.

[ https://issues.apache.org/jira/browse/SPARK-33762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33762: Assignee: (was: Apache Spark) > Bump commons-codec to latest version. >

[jira] [Commented] (SPARK-32526) Let sql/catalyst module tests pass for Scala 2.13

[ https://issues.apache.org/jira/browse/SPARK-32526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17248259#comment-17248259 ] Darcy Shen commented on SPARK-32526: ExpressionEncoderSuite still fails to work:

[jira] [Assigned] (SPARK-33653) DSv2: REFRESH TABLE should recache the table itself

[ https://issues.apache.org/jira/browse/SPARK-33653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33653: Assignee: (was: Apache Spark) > DSv2: REFRESH TABLE should recache the table itself

[jira] [Assigned] (SPARK-33653) DSv2: REFRESH TABLE should recache the table itself

[ https://issues.apache.org/jira/browse/SPARK-33653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33653: Assignee: Apache Spark > DSv2: REFRESH TABLE should recache the table itself >

[jira] [Commented] (SPARK-33653) DSv2: REFRESH TABLE should recache the table itself

[ https://issues.apache.org/jira/browse/SPARK-33653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17248257#comment-17248257 ] Apache Spark commented on SPARK-33653: -- User 'sunchao' has created a pull request for this issue:

[jira] [Commented] (SPARK-26345) Parquet support Column indexes

[ https://issues.apache.org/jira/browse/SPARK-26345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17248241#comment-17248241 ] Xinli Shang commented on SPARK-26345: - The Presto and Iceberg effort are not tied to each other. It

[jira] [Commented] (SPARK-26345) Parquet support Column indexes

[ https://issues.apache.org/jira/browse/SPARK-26345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17248236#comment-17248236 ] James R. Taylor commented on SPARK-26345: - Thanks for the update, [~sha...@uber.com]. I had just

[jira] [Resolved] (SPARK-33729) When refreshing cache, Spark should not use cached plan when recaching data

[ https://issues.apache.org/jira/browse/SPARK-33729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-33729. --- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 30699

[jira] [Assigned] (SPARK-33729) When refreshing cache, Spark should not use cached plan when recaching data

[ https://issues.apache.org/jira/browse/SPARK-33729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-33729: - Assignee: Chao Sun > When refreshing cache, Spark should not use cached plan when

[jira] [Commented] (SPARK-26345) Parquet support Column indexes

[ https://issues.apache.org/jira/browse/SPARK-26345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17248231#comment-17248231 ] Xinli Shang commented on SPARK-26345: - For the performance, there is an Eng Blog I found online

[jira] [Assigned] (SPARK-33764) Make state store maintenance interval as SQL config

[ https://issues.apache.org/jira/browse/SPARK-33764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33764: Assignee: L. C. Hsieh (was: Apache Spark) > Make state store maintenance interval as

[jira] [Assigned] (SPARK-33764) Make state store maintenance interval as SQL config

[ https://issues.apache.org/jira/browse/SPARK-33764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33764: Assignee: Apache Spark (was: L. C. Hsieh) > Make state store maintenance interval as

[jira] [Commented] (SPARK-33764) Make state store maintenance interval as SQL config

[ https://issues.apache.org/jira/browse/SPARK-33764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17248217#comment-17248217 ] Apache Spark commented on SPARK-33764: -- User 'viirya' has created a pull request for this issue:

[jira] [Commented] (SPARK-26345) Parquet support Column indexes

[ https://issues.apache.org/jira/browse/SPARK-26345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17248216#comment-17248216 ] James R. Taylor commented on SPARK-26345: - Any updates on this issue, [~zi]? Wouldn't column

[jira] [Created] (SPARK-33764) Make state store maintenance interval as SQL config

L. C. Hsieh created SPARK-33764: --- Summary: Make state store maintenance interval as SQL config Key: SPARK-33764 URL: https://issues.apache.org/jira/browse/SPARK-33764 Project: Spark Issue

[jira] [Commented] (SPARK-32617) Upgrade kubernetes client version to support latest minikube version.

[ https://issues.apache.org/jira/browse/SPARK-32617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17248168#comment-17248168 ] Attila Zsolt Piros commented on SPARK-32617: I will have PR for this soon. > Upgrade

[jira] [Commented] (SPARK-33731) Standardize exception types

[ https://issues.apache.org/jira/browse/SPARK-33731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17248156#comment-17248156 ] Shril Kumar commented on SPARK-33731: - Most welcome [~hyukjin.kwon]. :) > Standardize exception

[jira] [Commented] (SPARK-33730) Standardize warning types

[ https://issues.apache.org/jira/browse/SPARK-33730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17248155#comment-17248155 ] Shril Kumar commented on SPARK-33730: - [~hyukjin.kwon], thank you for your response. Looking forward

[jira] [Created] (SPARK-33763) Add metrics for better tracking of dynamic allocation

Holden Karau created SPARK-33763: Summary: Add metrics for better tracking of dynamic allocation Key: SPARK-33763 URL: https://issues.apache.org/jira/browse/SPARK-33763 Project: Spark Issue

[jira] [Commented] (SPARK-33752) Avoid the getSimpleMessage of AnalysisException adds semicolon repeatedly

[ https://issues.apache.org/jira/browse/SPARK-33752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17248121#comment-17248121 ] Apache Spark commented on SPARK-33752: -- User 'n-marion' has created a pull request for this issue:

[jira] [Updated] (SPARK-33762) Bump commons-codec to latest version.

[ https://issues.apache.org/jira/browse/SPARK-33762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Marion updated SPARK-33762: Description: Currently Spark pulls in commons-codec version 2.10 which was released 6

[jira] [Created] (SPARK-33762) Bump commons-codec to latest version.

Nicholas Marion created SPARK-33762: --- Summary: Bump commons-codec to latest version. Key: SPARK-33762 URL: https://issues.apache.org/jira/browse/SPARK-33762 Project: Spark Issue Type:

[jira] [Commented] (SPARK-22256) Introduce spark.mesos.driver.memoryOverhead

[ https://issues.apache.org/jira/browse/SPARK-22256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17248112#comment-17248112 ] Apache Spark commented on SPARK-22256: -- User 'dmcwhorter' has created a pull request for this

[jira] [Commented] (SPARK-22256) Introduce spark.mesos.driver.memoryOverhead

[ https://issues.apache.org/jira/browse/SPARK-22256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17248110#comment-17248110 ] Apache Spark commented on SPARK-22256: -- User 'dmcwhorter' has created a pull request for this

[jira] [Created] (SPARK-33761) [k8s] Support fetching driver and executor pod templates from HCFS

Xuzhou Yin created SPARK-33761: -- Summary: [k8s] Support fetching driver and executor pod templates from HCFS Key: SPARK-33761 URL: https://issues.apache.org/jira/browse/SPARK-33761 Project: Spark

[jira] [Commented] (SPARK-33695) Bump Jackson to 2.10.5 and databind to 2.10.5.1

[ https://issues.apache.org/jira/browse/SPARK-33695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17248107#comment-17248107 ] Nicholas Marion commented on SPARK-33695: - [~dongjoon] , As a security issue, would this

[jira] [Resolved] (SPARK-33576) PythonException: An exception was thrown from a UDF: 'OSError: Invalid IPC message: negative bodyLength'.

[ https://issues.apache.org/jira/browse/SPARK-33576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler resolved SPARK-33576. -- Resolution: Duplicate Going to resolve as a duplicate, but please reopen if you find it is

[jira] [Updated] (SPARK-33742) Throw PartitionsAlreadyExistException from HiveExternalCatalog.createPartitions()

[ https://issues.apache.org/jira/browse/SPARK-33742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-33742: -- Fix Version/s: (was: 3.2.0) 3.1.0 3.0.2

[jira] [Commented] (SPARK-33576) PythonException: An exception was thrown from a UDF: 'OSError: Invalid IPC message: negative bodyLength'.

[ https://issues.apache.org/jira/browse/SPARK-33576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17248064#comment-17248064 ] Bryan Cutler commented on SPARK-33576: -- [~darshats] I believe the only current workaround is to

[jira] [Comment Edited] (SPARK-33730) Standardize warning types

[ https://issues.apache.org/jira/browse/SPARK-33730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17248051#comment-17248051 ] Maciej Szymkiewicz edited comment on SPARK-33730 at 12/11/20, 5:30 PM:

[jira] [Comment Edited] (SPARK-33730) Standardize warning types

[ https://issues.apache.org/jira/browse/SPARK-33730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17248051#comment-17248051 ] Maciej Szymkiewicz edited comment on SPARK-33730 at 12/11/20, 5:29 PM:

[jira] [Commented] (SPARK-33730) Standardize warning types

[ https://issues.apache.org/jira/browse/SPARK-33730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17248051#comment-17248051 ] Maciej Szymkiewicz commented on SPARK-33730: {quote}Eh, I think Maciej Szymkiewicz is

[jira] [Created] (SPARK-33760) Extend Dynamic Partition Pruning Support to DataSources

Anoop Johnson created SPARK-33760: - Summary: Extend Dynamic Partition Pruning Support to DataSources Key: SPARK-33760 URL: https://issues.apache.org/jira/browse/SPARK-33760 Project: Spark

[jira] [Commented] (SPARK-32398) Upgrade to scalatest 3.2.0 for Scala 2.13.3 compatibility

[ https://issues.apache.org/jira/browse/SPARK-32398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17248021#comment-17248021 ] Steve Loughran commented on SPARK-32398: doesn't work with old versions though. I think I'll end

[jira] [Commented] (SPARK-33044) Add a Jenkins build and test job for Scala 2.13

[ https://issues.apache.org/jira/browse/SPARK-33044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17248013#comment-17248013 ] Shane Knapp commented on SPARK-33044: - done > Add a Jenkins build and test job for Scala 2.13 >

[jira] [Resolved] (SPARK-33746) Minikube is failing to start on research-jenkins-worker-05

[ https://issues.apache.org/jira/browse/SPARK-33746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shane Knapp resolved SPARK-33746. - Resolution: Fixed > Minikube is failing to start on research-jenkins-worker-05 >

[jira] [Assigned] (SPARK-33759) docker entrypoint should using `spark-class` for spark executor

[ https://issues.apache.org/jira/browse/SPARK-33759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33759: Assignee: (was: Apache Spark) > docker entrypoint should using `spark-class` for

[jira] [Commented] (SPARK-33759) docker entrypoint should using `spark-class` for spark executor

[ https://issues.apache.org/jira/browse/SPARK-33759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17248004#comment-17248004 ] Apache Spark commented on SPARK-33759: -- User 'dungdm93' has created a pull request for this issue:

[jira] [Commented] (SPARK-33759) docker entrypoint should using `spark-class` for spark executor

[ https://issues.apache.org/jira/browse/SPARK-33759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17248006#comment-17248006 ] Apache Spark commented on SPARK-33759: -- User 'dungdm93' has created a pull request for this issue:

[jira] [Assigned] (SPARK-33759) docker entrypoint should using `spark-class` for spark executor

[ https://issues.apache.org/jira/browse/SPARK-33759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33759: Assignee: Apache Spark > docker entrypoint should using `spark-class` for spark executor

[jira] [Commented] (SPARK-33044) Add a Jenkins build and test job for Scala 2.13

[ https://issues.apache.org/jira/browse/SPARK-33044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17248003#comment-17248003 ] Shane Knapp commented on SPARK-33044: - ah, ok...  i thought i just needed to run

[jira] [Updated] (SPARK-33759) docker entrypoint should using `spark-class` for spark executor

[ https://issues.apache.org/jira/browse/SPARK-33759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dung Dang Minh updated SPARK-33759: --- Summary: docker entrypoint should using `spark-class` for spark executor (was: spark

[jira] [Created] (SPARK-33759) spark executor should using `spark-class` in docker entrypoint

Dung Dang Minh created SPARK-33759: -- Summary: spark executor should using `spark-class` in docker entrypoint Key: SPARK-33759 URL: https://issues.apache.org/jira/browse/SPARK-33759 Project: Spark

[jira] [Commented] (SPARK-33746) Minikube is failing to start on research-jenkins-worker-05

[ https://issues.apache.org/jira/browse/SPARK-33746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247998#comment-17247998 ] Shane Knapp commented on SPARK-33746: - k8s master build passed:

[jira] [Resolved] (SPARK-33757) Fix the R dependencies build error on GitHub Actions and AppVeyor

[ https://issues.apache.org/jira/browse/SPARK-33757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-33757. -- Fix Version/s: 2.4.8 3.0.2 3.1.0 Resolution:

[jira] [Assigned] (SPARK-33526) Add config to control if cancel invoke interrupt task on thriftserver

[ https://issues.apache.org/jira/browse/SPARK-33526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-33526: Assignee: ulysses you > Add config to control if cancel invoke interrupt task on

[jira] [Resolved] (SPARK-33526) Add config to control if cancel invoke interrupt task on thriftserver

[ https://issues.apache.org/jira/browse/SPARK-33526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-33526. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 30481

[jira] [Created] (SPARK-33758) Prune unnecessary output partitioning when the attribute is not part of output.

Prakhar Jain created SPARK-33758: Summary: Prune unnecessary output partitioning when the attribute is not part of output. Key: SPARK-33758 URL: https://issues.apache.org/jira/browse/SPARK-33758

[jira] [Updated] (SPARK-33757) Fix the R dependencies build error on GitHub Actions and AppVeyor

[ https://issues.apache.org/jira/browse/SPARK-33757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-33757: --- Description: R dependencies build error happens now. The reason seems that usethis package

[jira] [Updated] (SPARK-33757) Fix the R dependencies build error on GitHub Actions and AppVeyor

[ https://issues.apache.org/jira/browse/SPARK-33757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-33757: --- Summary: Fix the R dependencies build error on GitHub Actions and AppVeyor (was: Fix the R

[jira] [Commented] (SPARK-33757) Fix the R dependencies build error on GitHub Actions

[ https://issues.apache.org/jira/browse/SPARK-33757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247930#comment-17247930 ] Apache Spark commented on SPARK-33757: -- User 'sarutak' has created a pull request for this issue:

[jira] [Assigned] (SPARK-33757) Fix the R dependencies build error on GitHub Actions

[ https://issues.apache.org/jira/browse/SPARK-33757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33757: Assignee: Kousuke Saruta (was: Apache Spark) > Fix the R dependencies build error on

[jira] [Assigned] (SPARK-33757) Fix the R dependencies build error on GitHub Actions

[ https://issues.apache.org/jira/browse/SPARK-33757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33757: Assignee: Apache Spark (was: Kousuke Saruta) > Fix the R dependencies build error on

[jira] [Created] (SPARK-33757) Fix the R dependencies build error on GitHub Actions

Kousuke Saruta created SPARK-33757: -- Summary: Fix the R dependencies build error on GitHub Actions Key: SPARK-33757 URL: https://issues.apache.org/jira/browse/SPARK-33757 Project: Spark

[jira] [Commented] (SPARK-33617) Avoid generating small files for INSERT INTO TABLE from VALUES

[ https://issues.apache.org/jira/browse/SPARK-33617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247926#comment-17247926 ] Apache Spark commented on SPARK-33617: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Commented] (SPARK-33617) Avoid generating small files for INSERT INTO TABLE from VALUES

[ https://issues.apache.org/jira/browse/SPARK-33617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247925#comment-17247925 ] Apache Spark commented on SPARK-33617: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Commented] (SPARK-33731) Standardize exception types

[ https://issues.apache.org/jira/browse/SPARK-33731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247923#comment-17247923 ] Hyukjin Kwon commented on SPARK-33731: -- Sure, thanks [~shril]. I have some thoughts on my mind. I

[jira] [Comment Edited] (SPARK-33730) Standardize warning types

[ https://issues.apache.org/jira/browse/SPARK-33730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247922#comment-17247922 ] Hyukjin Kwon edited comment on SPARK-33730 at 12/11/20, 1:17 PM: - Eh, I

[jira] [Commented] (SPARK-33730) Standardize warning types

[ https://issues.apache.org/jira/browse/SPARK-33730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247922#comment-17247922 ] Hyukjin Kwon commented on SPARK-33730: -- Eh, I think [~zero323] is already working on it(?). I will

[jira] [Assigned] (SPARK-33748) Support PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON environment variables

[ https://issues.apache.org/jira/browse/SPARK-33748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33748: Assignee: (was: Apache Spark) > Support PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON

[jira] [Commented] (SPARK-33748) Support PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON environment variables

[ https://issues.apache.org/jira/browse/SPARK-33748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247914#comment-17247914 ] Apache Spark commented on SPARK-33748: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-33748) Support PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON environment variables

[ https://issues.apache.org/jira/browse/SPARK-33748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33748: Assignee: Apache Spark > Support PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON environment

[jira] [Updated] (SPARK-33748) Support PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON environment variables

[ https://issues.apache.org/jira/browse/SPARK-33748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-33748: - Summary: Support PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON environment variables (was: Respect

[jira] [Assigned] (SPARK-33706) Require fully specified partition identifier in partitionExists()

[ https://issues.apache.org/jira/browse/SPARK-33706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-33706: --- Assignee: Maxim Gekk > Require fully specified partition identifier in partitionExists() >

[jira] [Resolved] (SPARK-33706) Require fully specified partition identifier in partitionExists()

[ https://issues.apache.org/jira/browse/SPARK-33706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-33706. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 30667

[jira] [Assigned] (SPARK-33755) Allow creating orc table when row format separator is defined

[ https://issues.apache.org/jira/browse/SPARK-33755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33755: Assignee: (was: Apache Spark) > Allow creating orc table when row format separator

[jira] [Commented] (SPARK-33755) Allow creating orc table when row format separator is defined

[ https://issues.apache.org/jira/browse/SPARK-33755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247894#comment-17247894 ] Apache Spark commented on SPARK-33755: -- User 'StefanXiepj' has created a pull request for this

[jira] [Assigned] (SPARK-33755) Allow creating orc table when row format separator is defined

[ https://issues.apache.org/jira/browse/SPARK-33755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33755: Assignee: (was: Apache Spark) > Allow creating orc table when row format separator

[jira] [Assigned] (SPARK-33755) Allow creating orc table when row format separator is defined

[ https://issues.apache.org/jira/browse/SPARK-33755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33755: Assignee: Apache Spark > Allow creating orc table when row format separator is defined >

[jira] [Assigned] (SPARK-33654) Migrate CACHE TABLE to new resolution framework

[ https://issues.apache.org/jira/browse/SPARK-33654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-33654: --- Assignee: Terry Kim > Migrate CACHE TABLE to new resolution framework >

[jira] [Resolved] (SPARK-33654) Migrate CACHE TABLE to new resolution framework

[ https://issues.apache.org/jira/browse/SPARK-33654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-33654. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 30598

[jira] [Commented] (SPARK-33635) Performance regression in Kafka read

[ https://issues.apache.org/jira/browse/SPARK-33635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247879#comment-17247879 ] David Wyles commented on SPARK-33635: - [~gsomogyi]  "try to turn off Kafka consumer caching. Apart

[jira] [Commented] (SPARK-33635) Performance regression in Kafka read

[ https://issues.apache.org/jira/browse/SPARK-33635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247878#comment-17247878 ] David Wyles commented on SPARK-33635: - "Since you're measuring speed I've ported the Kafka source

[jira] [Comment Edited] (SPARK-33635) Performance regression in Kafka read

[ https://issues.apache.org/jira/browse/SPARK-33635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247875#comment-17247875 ] David Wyles edited comment on SPARK-33635 at 12/11/20, 12:12 PM: - I'll

[jira] [Comment Edited] (SPARK-33635) Performance regression in Kafka read

[ https://issues.apache.org/jira/browse/SPARK-33635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247875#comment-17247875 ] David Wyles edited comment on SPARK-33635 at 12/11/20, 12:11 PM: - I'll

[jira] [Comment Edited] (SPARK-33635) Performance regression in Kafka read

[ https://issues.apache.org/jira/browse/SPARK-33635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247875#comment-17247875 ] David Wyles edited comment on SPARK-33635 at 12/11/20, 12:11 PM: - I'll

[jira] [Commented] (SPARK-33635) Performance regression in Kafka read

[ https://issues.apache.org/jira/browse/SPARK-33635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247875#comment-17247875 ] David Wyles commented on SPARK-33635: - I'll give all those a go and get back to you. > Performance

[jira] [Commented] (SPARK-33691) Support partition filters in ALTER TABLE DROP PARTITION

[ https://issues.apache.org/jira/browse/SPARK-33691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247870#comment-17247870 ] Apache Spark commented on SPARK-33691: -- User 'StefanXiepj' has created a pull request for this

[jira] [Commented] (SPARK-33691) Support partition filters in ALTER TABLE DROP PARTITION

[ https://issues.apache.org/jira/browse/SPARK-33691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247871#comment-17247871 ] Apache Spark commented on SPARK-33691: -- User 'StefanXiepj' has created a pull request for this

[jira] [Commented] (SPARK-33742) Throw PartitionsAlreadyExistException from HiveExternalCatalog.createPartitions()

[ https://issues.apache.org/jira/browse/SPARK-33742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247866#comment-17247866 ] Apache Spark commented on SPARK-33742: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Commented] (SPARK-33742) Throw PartitionsAlreadyExistException from HiveExternalCatalog.createPartitions()

[ https://issues.apache.org/jira/browse/SPARK-33742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247850#comment-17247850 ] Apache Spark commented on SPARK-33742: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Comment Edited] (SPARK-33730) Standardize warning types

[ https://issues.apache.org/jira/browse/SPARK-33730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247827#comment-17247827 ] Shril Kumar edited comment on SPARK-33730 at 12/11/20, 10:40 AM: -

[jira] [Commented] (SPARK-33730) Standardize warning types

[ https://issues.apache.org/jira/browse/SPARK-33730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247827#comment-17247827 ] Shril Kumar commented on SPARK-33730: - I was inspecting the use of the print statements in PySpark.

[jira] [Commented] (SPARK-33742) Throw PartitionsAlreadyExistException from HiveExternalCatalog.createPartitions()

[ https://issues.apache.org/jira/browse/SPARK-33742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247822#comment-17247822 ] Apache Spark commented on SPARK-33742: -- User 'MaxGekk' has created a pull request for this issue:

[jira] [Commented] (SPARK-33542) Group exception messages in catalyst/catalog

[ https://issues.apache.org/jira/browse/SPARK-33542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247818#comment-17247818 ] jiaan.geng commented on SPARK-33542: I'm working on. > Group exception messages in catalyst/catalog

[jira] [Updated] (SPARK-28367) Kafka connector infinite wait because metadata never updated

[ https://issues.apache.org/jira/browse/SPARK-28367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated SPARK-28367: -- Fix Version/s: 3.1.0 > Kafka connector infinite wait because metadata never updated >

[jira] [Closed] (SPARK-28367) Kafka connector infinite wait because metadata never updated

[ https://issues.apache.org/jira/browse/SPARK-28367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi closed SPARK-28367. - > Kafka connector infinite wait because metadata never updated >

[jira] [Resolved] (SPARK-28367) Kafka connector infinite wait because metadata never updated

[ https://issues.apache.org/jira/browse/SPARK-28367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi resolved SPARK-28367. --- Resolution: Fixed > Kafka connector infinite wait because metadata never updated >

[jira] [Commented] (SPARK-28367) Kafka connector infinite wait because metadata never updated

[ https://issues.apache.org/jira/browse/SPARK-28367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247813#comment-17247813 ] Gabor Somogyi commented on SPARK-28367: --- The issue solved in subtasks so closing this. > Kafka

[jira] [Assigned] (SPARK-33756) BytesToBytesMap's iterator hasNext method should be idempotent.

[ https://issues.apache.org/jira/browse/SPARK-33756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33756: Assignee: Apache Spark > BytesToBytesMap's iterator hasNext method should be idempotent.

[jira] [Commented] (SPARK-33756) BytesToBytesMap's iterator hasNext method should be idempotent.

[ https://issues.apache.org/jira/browse/SPARK-33756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247802#comment-17247802 ] Apache Spark commented on SPARK-33756: -- User 'advancedxy' has created a pull request for this

[jira] [Assigned] (SPARK-33756) BytesToBytesMap's iterator hasNext method should be idempotent.

[ https://issues.apache.org/jira/browse/SPARK-33756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33756: Assignee: (was: Apache Spark) > BytesToBytesMap's iterator hasNext method should be

[jira] [Resolved] (SPARK-33754) Update kubernetes/integration-tests/README.md to follow the default Hadoop profile updated

[ https://issues.apache.org/jira/browse/SPARK-33754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-33754. --- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 30726

[jira] [Updated] (SPARK-33755) Allow creating orc table when row format separator is defined

[ https://issues.apache.org/jira/browse/SPARK-33755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiepengjie updated SPARK-33755: --- Description: When creating table like this: {code:java} create table test_orc(c1 string) row format

  1   2   >