[jira] [Assigned] (SPARK-36150) Disable MiMa for Scala 2.13 artifacts

[ https://issues.apache.org/jira/browse/SPARK-36150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36150: Assignee: (was: Apache Spark) > Disable MiMa for Scala 2.13 artifacts > -

[jira] [Assigned] (SPARK-36150) Disable MiMa for Scala 2.13 artifacts

[ https://issues.apache.org/jira/browse/SPARK-36150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36150: Assignee: Apache Spark > Disable MiMa for Scala 2.13 artifacts >

[jira] [Commented] (SPARK-36150) Disable MiMa for Scala 2.13 artifacts

[ https://issues.apache.org/jira/browse/SPARK-36150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381091#comment-17381091 ] Apache Spark commented on SPARK-36150: -- User 'dongjoon-hyun' has created a pull req

[jira] [Updated] (SPARK-36150) Disable MiMa for Scala 2.13 artifacts

[ https://issues.apache.org/jira/browse/SPARK-36150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-36150: -- Affects Version/s: 3.2.0 > Disable MiMa for Scala 2.13 artifacts > ---

[jira] [Created] (SPARK-36151) Enable MiMa for Scala 2.13 artifacts after Spark 3.2.0 release

Dongjoon Hyun created SPARK-36151: - Summary: Enable MiMa for Scala 2.13 artifacts after Spark 3.2.0 release Key: SPARK-36151 URL: https://issues.apache.org/jira/browse/SPARK-36151 Project: Spark

[jira] [Created] (SPARK-36150) Disable MiMa for Scala 2.13 artifacts

Dongjoon Hyun created SPARK-36150: - Summary: Disable MiMa for Scala 2.13 artifacts Key: SPARK-36150 URL: https://issues.apache.org/jira/browse/SPARK-36150 Project: Spark Issue Type: Test

[jira] [Updated] (SPARK-36090) Support TimestampNTZType in expression Sequence

[ https://issues.apache.org/jira/browse/SPARK-36090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-36090: --- Summary: Support TimestampNTZType in expression Sequence (was: Support TimestampNTZ type in expres

[jira] [Updated] (SPARK-36149) dayofweek documentation for python and R

[ https://issues.apache.org/jira/browse/SPARK-36149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dominik Gehl updated SPARK-36149: - Description: Python and R documentation for {{dayofweek}} doesn't mention which integer correspo

[jira] [Commented] (SPARK-36149) dayofweek documentation for python and R

[ https://issues.apache.org/jira/browse/SPARK-36149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381081#comment-17381081 ] Apache Spark commented on SPARK-36149: -- User 'dominikgehl' has created a pull reque

[jira] [Assigned] (SPARK-36149) dayofweek documentation for python and R

[ https://issues.apache.org/jira/browse/SPARK-36149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36149: Assignee: (was: Apache Spark) > dayofweek documentation for python and R > --

[jira] [Commented] (SPARK-36149) dayofweek documentation for python and R

[ https://issues.apache.org/jira/browse/SPARK-36149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381078#comment-17381078 ] Apache Spark commented on SPARK-36149: -- User 'dominikgehl' has created a pull reque

[jira] [Assigned] (SPARK-36149) dayofweek documentation for python and R

[ https://issues.apache.org/jira/browse/SPARK-36149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36149: Assignee: Apache Spark > dayofweek documentation for python and R > -

[jira] [Created] (SPARK-36149) dayofweek documentation for python and R

Dominik Gehl created SPARK-36149: Summary: dayofweek documentation for python and R Key: SPARK-36149 URL: https://issues.apache.org/jira/browse/SPARK-36149 Project: Spark Issue Type: Improvem

[jira] [Commented] (SPARK-36037) Support ANSI SQL LOCALTIMESTAMP datetime value function

[ https://issues.apache.org/jira/browse/SPARK-36037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381061#comment-17381061 ] Apache Spark commented on SPARK-36037: -- User 'gengliangwang' has created a pull req

[jira] [Commented] (SPARK-29519) SHOW TBLPROPERTIES should look up catalog/table like v2 commands

[ https://issues.apache.org/jira/browse/SPARK-29519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381049#comment-17381049 ] Apache Spark commented on SPARK-29519: -- User 'Peng-Lei' has created a pull request

[jira] [Commented] (SPARK-29519) SHOW TBLPROPERTIES should look up catalog/table like v2 commands

[ https://issues.apache.org/jira/browse/SPARK-29519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381050#comment-17381050 ] Apache Spark commented on SPARK-29519: -- User 'Peng-Lei' has created a pull request

[jira] [Updated] (SPARK-36134) jackson-databind RCE vulnerability

[ https://issues.apache.org/jira/browse/SPARK-36134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumit updated SPARK-36134: -- Summary: jackson-databind RCE vulnerability (was: jackson-databind RCE vulnerability [Need to upgrade to 2.9.

[jira] [Updated] (SPARK-36134) jackson-databind RCE vulnerability [Need to upgrade to 2.9.3.1]

[ https://issues.apache.org/jira/browse/SPARK-36134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumit updated SPARK-36134: -- Attachment: Screenshot 2021-07-15 at 1.00.55 PM.png > jackson-databind RCE vulnerability [Need to upgrade to 2

[jira] [Commented] (SPARK-36134) jackson-databind RCE vulnerability [Need to upgrade to 2.9.3.1]

[ https://issues.apache.org/jira/browse/SPARK-36134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381038#comment-17381038 ] Sumit commented on SPARK-36134: --- this is reported by the security team [Open-Source Scan -

[jira] [Updated] (SPARK-36134) jackson-databind RCE vulnerability [Need to upgrade to 2.9.3.1]

[ https://issues.apache.org/jira/browse/SPARK-36134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumit updated SPARK-36134: -- Attachment: Screenshot 2021-07-15 at 1.00.55 PM.png > jackson-databind RCE vulnerability [Need to upgrade to 2

[jira] [Updated] (SPARK-36134) jackson-databind RCE vulnerability [Need to upgrade to 2.9.3.1]

[ https://issues.apache.org/jira/browse/SPARK-36134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumit updated SPARK-36134: -- Attachment: (was: Screenshot 2021-07-15 at 1.00.55 PM.png) > jackson-databind RCE vulnerability [Need to u

[jira] [Updated] (SPARK-36134) jackson-databind RCE vulnerability [Need to upgrade to 2.9.3.1]

[ https://issues.apache.org/jira/browse/SPARK-36134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumit updated SPARK-36134: -- Attachment: (was: Screenshot 2021-07-15 at 1.00.55 PM.png) > jackson-databind RCE vulnerability [Need to u

[jira] [Updated] (SPARK-36134) jackson-databind RCE vulnerability [Need to upgrade to 2.9.3.1]

[ https://issues.apache.org/jira/browse/SPARK-36134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sumit updated SPARK-36134: -- Attachment: Screenshot 2021-07-15 at 1.00.55 PM.png > jackson-databind RCE vulnerability [Need to upgrade to 2

[jira] [Created] (SPARK-36148) Missing validation of regexp_replace inputs

Linhong Liu created SPARK-36148: --- Summary: Missing validation of regexp_replace inputs Key: SPARK-36148 URL: https://issues.apache.org/jira/browse/SPARK-36148 Project: Spark Issue Type: Task

[jira] [Commented] (SPARK-35334) Spark should be more resilient to intermittent K8s flakiness

[ https://issues.apache.org/jira/browse/SPARK-35334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381013#comment-17381013 ] Dongjoon Hyun commented on SPARK-35334: --- Thank you for updating, [~attilapiros].

[jira] [Updated] (SPARK-36076) [SQL] ArrayIndexOutOfBounds in CAST string to timestamp

[ https://issues.apache.org/jira/browse/SPARK-36076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-36076: -- Affects Version/s: 3.0.3 3.1.2 > [SQL] ArrayIndexOutOfBounds in CAST st

[jira] [Resolved] (SPARK-36076) [SQL] ArrayIndexOutOfBounds in CAST string to timestamp

[ https://issues.apache.org/jira/browse/SPARK-36076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-36076. --- Fix Version/s: 3.0.4 Resolution: Fixed Issue resolved by pull request 33325 [https://

[jira] [Updated] (SPARK-36076) [SQL] ArrayIndexOutOfBounds in CAST string to timestamp

[ https://issues.apache.org/jira/browse/SPARK-36076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-36076: -- Fix Version/s: 3.1.3 > [SQL] ArrayIndexOutOfBounds in CAST string to timestamp > -

[jira] [Commented] (SPARK-36147) [SQL] - log level should be warning if files not found in BasicWriteStatsTracker

[ https://issues.apache.org/jira/browse/SPARK-36147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381010#comment-17381010 ] Apache Spark commented on SPARK-36147: -- User 'tooptoop4' has created a pull request

[jira] [Assigned] (SPARK-36147) [SQL] - log level should be warning if files not found in BasicWriteStatsTracker

[ https://issues.apache.org/jira/browse/SPARK-36147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36147: Assignee: (was: Apache Spark) > [SQL] - log level should be warning if files not foun

[jira] [Commented] (SPARK-36147) [SQL] - log level should be warning if files not found in BasicWriteStatsTracker

[ https://issues.apache.org/jira/browse/SPARK-36147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381008#comment-17381008 ] Apache Spark commented on SPARK-36147: -- User 'tooptoop4' has created a pull request

[jira] [Assigned] (SPARK-36147) [SQL] - log level should be warning if files not found in BasicWriteStatsTracker

[ https://issues.apache.org/jira/browse/SPARK-36147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36147: Assignee: Apache Spark > [SQL] - log level should be warning if files not found in > Bas

[jira] [Created] (SPARK-36147) [SQL] - log level should be warning if files not found in BasicWriteStatsTracker

t oo created SPARK-36147: Summary: [SQL] - log level should be warning if files not found in BasicWriteStatsTracker Key: SPARK-36147 URL: https://issues.apache.org/jira/browse/SPARK-36147 Project: Spark

[jira] [Commented] (SPARK-36145) Remove Python 3.6 support in codebase and CI

[ https://issues.apache.org/jira/browse/SPARK-36145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381006#comment-17381006 ] Dongjoon Hyun commented on SPARK-36145: --- Thank you, [~hyukjin.kwon]! > Remove Pyt

[jira] [Updated] (SPARK-36139) Remove Python 3.6 from `pyspark` GitHub Action job

[ https://issues.apache.org/jira/browse/SPARK-36139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-36139: -- Summary: Remove Python 3.6 from `pyspark` GitHub Action job (was: Remove Python 3.6 from `bui

[jira] [Commented] (SPARK-36090) Support TimestampNTZ type in expression Sequence

[ https://issues.apache.org/jira/browse/SPARK-36090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381005#comment-17381005 ] jiaan.geng commented on SPARK-36090: I'm working on. > Support TimestampNTZ type i

[jira] [Assigned] (SPARK-36139) Remove Python 3.6 from `build and test` GitHub Action job

[ https://issues.apache.org/jira/browse/SPARK-36139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-36139: - Assignee: Dongjoon Hyun > Remove Python 3.6 from `build and test` GitHub Action job > -

[jira] [Resolved] (SPARK-36139) Remove Python 3.6 from `build and test` GitHub Action job

[ https://issues.apache.org/jira/browse/SPARK-36139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-36139. --- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 33349 [https://

[jira] [Updated] (SPARK-36094) Group SQL component error messages in Spark error class JSON file

[ https://issues.apache.org/jira/browse/SPARK-36094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Feng updated SPARK-36094: --- Description: To improve auditing, reduce duplication, and improve quality of error messages thrown

[jira] [Updated] (SPARK-36106) Refactor a few query compilation errors to use error classes

[ https://issues.apache.org/jira/browse/SPARK-36106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Feng updated SPARK-36106: --- Description: Refactor some exceptions in [QueryCompilationErrors|https://github.com/apache/spark/bl

[jira] [Commented] (SPARK-36146) Upgrade Python version from 3.6 to higher version in GitHub linter

[ https://issues.apache.org/jira/browse/SPARK-36146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17380988#comment-17380988 ] Hyukjin Kwon commented on SPARK-36146: -- I'll work on this. FYI [~ueshin] [~dongjoon

[jira] [Updated] (SPARK-36106) Add subset of error classes to QueryCompilationErrors

[ https://issues.apache.org/jira/browse/SPARK-36106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Feng updated SPARK-36106: --- Summary: Add subset of error classes to QueryCompilationErrors (was: Add error classes to QueryComp

[jira] [Updated] (SPARK-36146) Upgrade Python version from 3.6 to higher version in GitHub linter

[ https://issues.apache.org/jira/browse/SPARK-36146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-36146: - Description: MyPy checks fails with higher Python versions. For example, with Python 3.8: {code

[jira] [Created] (SPARK-36146) Upgrade Python version from 3.6 to higher version in GitHub linter

Hyukjin Kwon created SPARK-36146: Summary: Upgrade Python version from 3.6 to higher version in GitHub linter Key: SPARK-36146 URL: https://issues.apache.org/jira/browse/SPARK-36146 Project: Spark

[jira] [Resolved] (SPARK-36144) Use Python 3.9 in `run-pip-tests` conda environment

[ https://issues.apache.org/jira/browse/SPARK-36144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36144. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 33351 [https://gi

[jira] [Assigned] (SPARK-36144) Use Python 3.9 in `run-pip-tests` conda environment

[ https://issues.apache.org/jira/browse/SPARK-36144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36144: Assignee: Dongjoon Hyun > Use Python 3.9 in `run-pip-tests` conda environment > -

[jira] [Commented] (SPARK-34952) Aggregate (Min/Max/Count) push down for Parquet

[ https://issues.apache.org/jira/browse/SPARK-34952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17380963#comment-17380963 ] Apache Spark commented on SPARK-34952: -- User 'huaxingao' has created a pull request

[jira] [Commented] (SPARK-34952) Aggregate (Min/Max/Count) push down for Parquet

[ https://issues.apache.org/jira/browse/SPARK-34952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17380961#comment-17380961 ] Apache Spark commented on SPARK-34952: -- User 'huaxingao' has created a pull request

[jira] [Resolved] (SPARK-36106) Add error classes to QueryCompilationErrors

[ https://issues.apache.org/jira/browse/SPARK-36106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36106. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33309 [https://gi

[jira] [Assigned] (SPARK-36106) Add error classes to QueryCompilationErrors

[ https://issues.apache.org/jira/browse/SPARK-36106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36106: Assignee: Karen Feng > Add error classes to QueryCompilationErrors >

[jira] [Commented] (SPARK-35810) Deprecate ps.broadcast API

[ https://issues.apache.org/jira/browse/SPARK-35810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17380956#comment-17380956 ] Haejoon Lee commented on SPARK-35810: - I'm working on this > Deprecate ps.broadcast

[jira] [Commented] (SPARK-36145) Remove Python 3.6 support in codebase and CI

[ https://issues.apache.org/jira/browse/SPARK-36145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17380934#comment-17380934 ] Hyukjin Kwon commented on SPARK-36145: -- cc [~ueshin] [~itholic] [~XinrongM] [~smile

[jira] [Updated] (SPARK-36145) Remove Python 3.6 support in codebase and CI

[ https://issues.apache.org/jira/browse/SPARK-36145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-36145: - Target Version/s: 3.3.0 > Remove Python 3.6 support in codebase and CI > ---

[jira] [Commented] (SPARK-36145) Remove Python 3.6 support in codebase and CI

[ https://issues.apache.org/jira/browse/SPARK-36145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17380933#comment-17380933 ] Hyukjin Kwon commented on SPARK-36145: -- [~dongjoon], I just made an umbrella ticket

[jira] [Updated] (SPARK-36139) Remove Python 3.6 from `build and test` GitHub Action job

[ https://issues.apache.org/jira/browse/SPARK-36139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-36139: - Parent: SPARK-36145 Issue Type: Sub-task (was: Test) > Remove Python 3.6 from `build an

[jira] [Updated] (SPARK-36145) Remove Python 3.6 support

[ https://issues.apache.org/jira/browse/SPARK-36145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-36145: - Description: Python 3.6 is deprecated via SPARK-35938 at Apache Spark 3.2. We should remove it

[jira] [Updated] (SPARK-36145) Remove Python 3.6 support in codebase and CI

[ https://issues.apache.org/jira/browse/SPARK-36145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-36145: - Summary: Remove Python 3.6 support in codebase and CI (was: Remove Python 3.6 support) > Remov

[jira] [Updated] (SPARK-36144) Use Python 3.9 in `run-pip-tests` conda environment

[ https://issues.apache.org/jira/browse/SPARK-36144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-36144: - Parent: SPARK-36145 Issue Type: Sub-task (was: Test) > Use Python 3.9 in `run-pip-tests

[jira] [Created] (SPARK-36145) Remove Python 3.6 support

Hyukjin Kwon created SPARK-36145: Summary: Remove Python 3.6 support Key: SPARK-36145 URL: https://issues.apache.org/jira/browse/SPARK-36145 Project: Spark Issue Type: Improvement C

[jira] [Assigned] (SPARK-36144) Use Python 3.9 in `run-pip-tests` conda environment

[ https://issues.apache.org/jira/browse/SPARK-36144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36144: Assignee: (was: Apache Spark) > Use Python 3.9 in `run-pip-tests` conda environment >

[jira] [Assigned] (SPARK-36144) Use Python 3.9 in `run-pip-tests` conda environment

[ https://issues.apache.org/jira/browse/SPARK-36144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36144: Assignee: Apache Spark > Use Python 3.9 in `run-pip-tests` conda environment > --

[jira] [Commented] (SPARK-36144) Use Python 3.9 in `run-pip-tests` conda environment

[ https://issues.apache.org/jira/browse/SPARK-36144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17380925#comment-17380925 ] Apache Spark commented on SPARK-36144: -- User 'dongjoon-hyun' has created a pull req

[jira] [Commented] (SPARK-36144) Use Python 3.9 in `run-pip-tests` conda environment

[ https://issues.apache.org/jira/browse/SPARK-36144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17380924#comment-17380924 ] Apache Spark commented on SPARK-36144: -- User 'dongjoon-hyun' has created a pull req

[jira] [Updated] (SPARK-36144) Use Python 3.9 in `run-pip-tests` conda environment

[ https://issues.apache.org/jira/browse/SPARK-36144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-36144: -- Summary: Use Python 3.9 in `run-pip-tests` conda environment (was: Use Python 3.9 in `run-pip

[jira] [Created] (SPARK-36144) Use Python 3.9 in `run-pip-tests` conda

Dongjoon Hyun created SPARK-36144: - Summary: Use Python 3.9 in `run-pip-tests` conda Key: SPARK-36144 URL: https://issues.apache.org/jira/browse/SPARK-36144 Project: Spark Issue Type: Test

[jira] [Commented] (SPARK-36136) Move PruneFileSourcePartitionsSuite out of org.apache.spark.sql.hive

[ https://issues.apache.org/jira/browse/SPARK-36136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17380911#comment-17380911 ] Apache Spark commented on SPARK-36136: -- User 'sunchao' has created a pull request f

[jira] [Assigned] (SPARK-36136) Move PruneFileSourcePartitionsSuite out of org.apache.spark.sql.hive

[ https://issues.apache.org/jira/browse/SPARK-36136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36136: Assignee: (was: Apache Spark) > Move PruneFileSourcePartitionsSuite out of org.apache

[jira] [Assigned] (SPARK-36136) Move PruneFileSourcePartitionsSuite out of org.apache.spark.sql.hive

[ https://issues.apache.org/jira/browse/SPARK-36136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36136: Assignee: Apache Spark > Move PruneFileSourcePartitionsSuite out of org.apache.spark.sql.

[jira] [Commented] (SPARK-36136) Move PruneFileSourcePartitionsSuite out of org.apache.spark.sql.hive

[ https://issues.apache.org/jira/browse/SPARK-36136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17380910#comment-17380910 ] Apache Spark commented on SPARK-36136: -- User 'sunchao' has created a pull request f

[jira] [Comment Edited] (SPARK-36031) Keep same behavior with pandas for operations of series with nan

[ https://issues.apache.org/jira/browse/SPARK-36031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17380902#comment-17380902 ] Xinrong Meng edited comment on SPARK-36031 at 7/14/21, 11:52 PM: -

[jira] [Commented] (SPARK-36031) Keep same behavior with pandas for operations of series with nan

[ https://issues.apache.org/jira/browse/SPARK-36031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17380902#comment-17380902 ] Xinrong Meng commented on SPARK-36031: -- Hi [~yikunkero], I dispatched the examples

[jira] [Comment Edited] (SPARK-36031) Keep same behavior with pandas for operations of series with nan

[ https://issues.apache.org/jira/browse/SPARK-36031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17380902#comment-17380902 ] Xinrong Meng edited comment on SPARK-36031 at 7/14/21, 11:51 PM: -

[jira] [Created] (SPARK-36143) Adjust astype of Series of ExtensionDtype to follow pandas

Xinrong Meng created SPARK-36143: Summary: Adjust astype of Series of ExtensionDtype to follow pandas Key: SPARK-36143 URL: https://issues.apache.org/jira/browse/SPARK-36143 Project: Spark Is

[jira] [Updated] (SPARK-36031) Keep same behavior with pandas for operations of series with nan

[ https://issues.apache.org/jira/browse/SPARK-36031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-36031: - Description: There are many operations for series containing nan values that don't follow pandas

[jira] [Created] (SPARK-36142) Adjust exponentiation between ExtentionDtypes and bools to follow pandas

Xinrong Meng created SPARK-36142: Summary: Adjust exponentiation between ExtentionDtypes and bools to follow pandas Key: SPARK-36142 URL: https://issues.apache.org/jira/browse/SPARK-36142 Project: Spa

[jira] [Updated] (SPARK-36031) Keep same behavior with pandas for operations of series with nan

[ https://issues.apache.org/jira/browse/SPARK-36031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-36031: - Description: There are many operations for series doen't follow the pandas, such as: ==

[jira] [Created] (SPARK-36141) Support arithmetic operations of Series containing Decimal(np.nan)

Xinrong Meng created SPARK-36141: Summary: Support arithmetic operations of Series containing Decimal(np.nan) Key: SPARK-36141 URL: https://issues.apache.org/jira/browse/SPARK-36141 Project: Spark

[jira] [Updated] (SPARK-36031) Keep same behavior with pandas for operations of series with nan

[ https://issues.apache.org/jira/browse/SPARK-36031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-36031: - Description: There are many operations for series doen't follow the pandas, such as: ==

[jira] [Created] (SPARK-36140) Replace DataTypeOps tests that have operations on different Series

Xinrong Meng created SPARK-36140: Summary: Replace DataTypeOps tests that have operations on different Series Key: SPARK-36140 URL: https://issues.apache.org/jira/browse/SPARK-36140 Project: Spark

[jira] [Updated] (SPARK-36139) Remove Python 3.6 from `build and test` GitHub Action job

[ https://issues.apache.org/jira/browse/SPARK-36139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-36139: -- Priority: Minor (was: Major) > Remove Python 3.6 from `build and test` GitHub Action job > --

[jira] [Assigned] (SPARK-36139) Remove Python 3.6 from `build and test` GitHub Action job

[ https://issues.apache.org/jira/browse/SPARK-36139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36139: Assignee: Apache Spark > Remove Python 3.6 from `build and test` GitHub Action job >

[jira] [Commented] (SPARK-36139) Remove Python 3.6 from `build and test` GitHub Action job

[ https://issues.apache.org/jira/browse/SPARK-36139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17380879#comment-17380879 ] Apache Spark commented on SPARK-36139: -- User 'dongjoon-hyun' has created a pull req

[jira] [Commented] (SPARK-36139) Remove Python 3.6 from `build and test` GitHub Action job

[ https://issues.apache.org/jira/browse/SPARK-36139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17380880#comment-17380880 ] Apache Spark commented on SPARK-36139: -- User 'dongjoon-hyun' has created a pull req

[jira] [Assigned] (SPARK-36139) Remove Python 3.6 from `build and test` GitHub Action job

[ https://issues.apache.org/jira/browse/SPARK-36139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36139: Assignee: (was: Apache Spark) > Remove Python 3.6 from `build and test` GitHub Action

[jira] [Created] (SPARK-36139) Remove Python 3.6 from `build and test` GitHub Action job

Dongjoon Hyun created SPARK-36139: - Summary: Remove Python 3.6 from `build and test` GitHub Action job Key: SPARK-36139 URL: https://issues.apache.org/jira/browse/SPARK-36139 Project: Spark I

[jira] [Updated] (SPARK-36138) Support for Slowly Changing Dimension Build Or Data Updates

[ https://issues.apache.org/jira/browse/SPARK-36138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nagharajan Raghavendran updated SPARK-36138: Summary: Support for Slowly Changing Dimension Build Or Data Updates (was:

[jira] [Resolved] (SPARK-36125) Implement non-equality comparison operators between two Categoricals

[ https://issues.apache.org/jira/browse/SPARK-36125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-36125. --- Fix Version/s: 3.2.0 Assignee: Xinrong Meng Resolution: Fixed Issue resolved

[jira] [Created] (SPARK-36138) Support for Slowly Changing Dimension Build

Nagharajan Raghavendran created SPARK-36138: --- Summary: Support for Slowly Changing Dimension Build Key: SPARK-36138 URL: https://issues.apache.org/jira/browse/SPARK-36138 Project: Spark

[jira] [Updated] (SPARK-36137) HiveShim always fallback to getAllPartitionsOf regardless of whether directSQL is enabled in remote HMS

[ https://issues.apache.org/jira/browse/SPARK-36137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated SPARK-36137: - Description: At the moment {{getPartitionsByFilter}} in Hive shim only fallback to use {{getAllPartitio

[jira] [Created] (SPARK-36137) HiveShim always fallback to getAllPartitionsOf regardless of whether directSQL is enabled in remote HMS

Chao Sun created SPARK-36137: Summary: HiveShim always fallback to getAllPartitionsOf regardless of whether directSQL is enabled in remote HMS Key: SPARK-36137 URL: https://issues.apache.org/jira/browse/SPARK-36137

[jira] [Assigned] (SPARK-36069) spark function from_json should output field name, field type and field value when FAILFAST mode throw exception

[ https://issues.apache.org/jira/browse/SPARK-36069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-36069: Assignee: Max Gekk > spark function from_json should output field name, field type and field valu

[jira] [Resolved] (SPARK-36069) spark function from_json should output field name, field type and field value when FAILFAST mode throw exception

[ https://issues.apache.org/jira/browse/SPARK-36069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-36069. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 33297 [https://github.com

[jira] [Created] (SPARK-36136) Move PruneFileSourcePartitionsSuite out of org.apache.spark.sql.hive

Chao Sun created SPARK-36136: Summary: Move PruneFileSourcePartitionsSuite out of org.apache.spark.sql.hive Key: SPARK-36136 URL: https://issues.apache.org/jira/browse/SPARK-36136 Project: Spark

[jira] [Assigned] (SPARK-35639) Add metrics about coalesced partitions to CustomShuffleReader in AQE

[ https://issues.apache.org/jira/browse/SPARK-35639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35639: Assignee: Apache Spark (was: Eugene Koifman) > Add metrics about coalesced partitions to

[jira] [Assigned] (SPARK-35639) Add metrics about coalesced partitions to CustomShuffleReader in AQE

[ https://issues.apache.org/jira/browse/SPARK-35639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35639: Assignee: Eugene Koifman (was: Apache Spark) > Add metrics about coalesced partitions to

[jira] [Commented] (SPARK-36128) CatalogFileIndex.filterPartitions should respect spark.sql.hive.metastorePartitionPruning

[ https://issues.apache.org/jira/browse/SPARK-36128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17380737#comment-17380737 ] Apache Spark commented on SPARK-36128: -- User 'sunchao' has created a pull request f

[jira] [Assigned] (SPARK-36128) CatalogFileIndex.filterPartitions should respect spark.sql.hive.metastorePartitionPruning

[ https://issues.apache.org/jira/browse/SPARK-36128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36128: Assignee: (was: Apache Spark) > CatalogFileIndex.filterPartitions should respect > s

[jira] [Commented] (SPARK-36128) CatalogFileIndex.filterPartitions should respect spark.sql.hive.metastorePartitionPruning

[ https://issues.apache.org/jira/browse/SPARK-36128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17380736#comment-17380736 ] Apache Spark commented on SPARK-36128: -- User 'sunchao' has created a pull request f

[jira] [Assigned] (SPARK-36128) CatalogFileIndex.filterPartitions should respect spark.sql.hive.metastorePartitionPruning

[ https://issues.apache.org/jira/browse/SPARK-36128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36128: Assignee: Apache Spark > CatalogFileIndex.filterPartitions should respect > spark.sql.hi

[jira] [Assigned] (SPARK-36034) Incorrect datetime filter when reading Parquet files written in legacy mode

[ https://issues.apache.org/jira/browse/SPARK-36034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36034: Assignee: Apache Spark > Incorrect datetime filter when reading Parquet files written in

  1   2   >